Application Number: 10/078,299 
Filing/Receipt Date: 02/20/2002 
First Named Applicant: Ying Liu 
Name of Examiner: Rosario, Dennis 

Name of Participants: Dennis Rosario (examiner), Ying Liu (applicant) 
Confirmation No. 3371 

Provisional Application Number 60/296,245 
Provisional Filing Date 06/06/2001 

Total Claims 28 4 



1 



Attrasoft Image Retrieval 



Inventor: Ying Liu, Savannah, GA 



ABSTRACT 4 
TECHNICAL FIELD 5 

BACKGROUND OF THE INVENTION f> 

TECHNICAL BACKGROUND 9 

1. Images ........................""9 

2. Mapping 11 

3. Markov Chain and Pattern Classification H 

4. Markov Chain and Neural Network 12 

5. Artificial Neural Net 12 

SUMMARY OF THE INVENTION 1 5 

1. Process J 15 

2. Parameters m 23 

3. System Layout .........................25 

4. Algorithms .........26 

5. Components and Application-Programming Interface 31 

BRIEF DESC RIPTION OF VIEWS OF THE DRAWING 32 

DETAILED DESCRIPTI ON OF THE DISCLOSED EMBODYMENT 36 

Preferred Embodiment of the Search System 36 

About This Invention # 36 

Overview of the ABM A l gorithm Process ...................37 



2 



Overview of the APN A l gorithm Process 40 

User Interface Layer of software for implementation of ABM and APN Algorithms 43 

Presentation Layer of software for implementation of ABM and APN Algorithms 51 

ABM Layer of software for implementation of ABM and APN Algorithms 52 

I V I- AP I (Imago Vorification and I dentification App l ication Programming Interface) 63 

Enro l lment t ................ 66 

1:N Match i ng 67 

N:N Matching ...........68 

P a rameters 69 

SAMPLE IMPLEMENTATION 70 

Attrasoft ImageFinder 4.9 70 

Attrasoft PolyApplet 5.0 73 

Attrasoft TransApplet 5.1 74 

CLAIMS 76 

APPENDIX A. ACCOMPANYING APPLICATION DOCUMENT AND CD 86 

Figure 1 94 

Figure 2 95 

Figure 3 96 

Figure 4 97 

Figure 5 93 

Figure 6 99 

Figure 7 100 

Figure 8 101 

Figure 9 102 

Figure 10 !03 

Figure 11 10 4 

Figure 12 105 

Figure 13 106 

Figure 14 10 7 

Figure 15 108 

Figure 16. {a) 109 

Figure ±6 17 110 

Figure 178 m 

Figure 189 112 



3 



Abstract 



A system, methods, and algorithm for content-based image retrieval and recognition system, 
useful in all types images and image formats. An image(s) or an image segment(s), which is 
specified by the user in two clicks (the first in the upper-left corner and the second in the bottom- 
right corner), specifies the content-based sample. The sample image(s) is used to teach the 
system what to look for via the ABM (Attrasoft Boltzmann Machine) algorithm and APN 
(Attrasoft PolyNet) algorithm; the system then searches through one or many directories, which 
is specified by the user, and presents the search research results. Tho search result consists of 
pairs match e d imago and a W e ight (score) The search result consists of pairs of matched images 
and its Weight (score), which specifies the similarity between the sample and matching images. 
These weights are also being used to classify images in the cases of the classification problem. 
The users are able to view the retrieved images in the result via a single click. Wh e n th e 
algorithm is implement e d as a software component, th e system integration will follow th e 
sp e cification of tho "Attrasoft Imag e Verification and Id e ntification Application Programming 
Intorfaco (IVI API)". 
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TECHNICAL FIELD 



The present invention relates generally to image retrieval and image recognition, and more 
particularly related to a system, methods, and algorithms of content-based image retrieval and 
recognition system. Within such a system, the image(s) to be retrieved/recognized is not 
preprocessed with the association of key words (meta-data). This system allows the user of an 
image retrieval/recognition system, such as software together with a computer, network server, 
or web server etc, to define a searching criteria by using an image(s), a segment of an image(s), a 
directory containing images or combinations of the above. This system will return the result, 
which contains pairs of the matched image and similarity. The user can see the matched images 
in a single click. 

This invention can be used in image verification (1-to-l matching, binary output: match/no 
match 1:1 matching, binary output: yes/no ), image identification (1 -to-many matching, single 
output to i ndicate a classification 1 :N matching, single output to indicate a classification ), image 
search or retrieval ( 1 -to-many matching, multiple output 1 :N matching, multiple output ), and 
image classification (many-to-1 or manv-to-many matching N:l or N:N matching ). Fee 
simplicity, w e will only us e th e word, retrieval. 
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BACKGROUND OF THE INVENTION 

In certain types of content-based images retrieval/recognition systems, the central task of the 
management system is to retrieve images that meet some specified constraints. 



Most image-retrieval methods are limited to the keyword-based approach via database queries. 
There are also methods that generate keyword-based via images (Example, LTU Technologies), 
which is fundamentally database queries based on computer-generated keywords. In this 
approach, keywords and the images together form a record in a table. The retrieval is based on 
the keywords in much the same way as the relational database. (Example: Microsoft Access). 

The current invention does not use keywords, neither manually generated keywords or keywords 
generated via some algorithms. The current invention consists of a set of algorithms. 

The user operation is generally divided into two phases: the learning phase and the 
search/recognition phase. In the learning phase, various types of processes, such as image 
preprocessing and image filtering are applied to the images. Then the images are sent to a 
recognition module to teach the module the characteristics of the image. The learning module 
can use various algorithms to learn the sample image £s). In the search/retrieval phase, the 
recognition module decides the classification of an image in a search directory or a search 
database. 

A very small number of commercially available products exist which perform content-based 
image retrieval. 



6 



Informix Internet Foundation.2000 is an object-relational database management system 
(ORDBMS), which supports non-alphanumeric data types (objects). IIF2000 supports several 
DataBlade modules including the Excalibur Image DataBlade module to extend its retrieval 
capabilities. DataBlade modules are server extensions that are integrated into the core of the 
database engine. The Excalibur Image DataBlade is based on technology from Excalibur 
Technologies Corporation, and is co-developed and co-supported by Informix and Excalibur. 
The core of the DataBlade is the Excalibur Visual retrievalWare SDK. The Image DataBlade 
module provides image storage, retrieval, and feature management for digital image data. This 
includes image manipulation, I/O routines, and feature extraction to store and retrieve images by 
their visual contents. An Informix database can be queried by aspect ratio, brightness, global 
colour, local colour, shape, and texture attributes. An evaluation copy of IIF2000 and the 
Excalibur Image DataBlade module can be downloaded from Informix 
www.informix.com/evaluato/ . 

IMatch is a content-based image retrieval system developed for the Windows operating system. 
The software was developed by Mario M. Westphal and is available under a shareware license. 
IMatch can query an image database by the following matching features: colour similarity, 
colour and shape (Quick), colour and shape (Fuzzy), colour percentage, and colour distribution. 
A folly functional 30-day evaluation copy is available for users to assess the software's 
capabilities and can be downloaded from Mario M. Westphal's web site. 
wvw.mwlabo.do/download.htm The shareware version has a 2000 limit on the number of images 
that can be added to a database. A new version of the software was released on the 18th February 
2001. 
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The Oracle8/ Enterprise Server is an object relational database management system that includes 
integral support for BLOBs. This provides the basis for adding complex objects, such as digital 
images, to Oracle databases. The Enterprise release of the Oracle database server includes the 
Visual Information retrieval (VIR) data cartridge developed by Virage Inc. OVIR is an extension 
to Oracle8/ Enterprise Server that provides image storage, content-based retrieval, and format 
conversion capabilities through an object type. An Oracle database can be queried by global 
color, local color, shape, and texture attributes. An evaluation copy of the Oracle8/ Enterprise 
Server can be downloaded from Oracle, Inc otn.oraclo.com . 
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TECHNICAL BACKGROUND 



The concepts in this section are well known and they are not a part of this invention. They are 
listed to facilitate t he understanding of the rest of this invention because these concepts are 
generally not grouped together in any textbooks or publications. In particular, the various spaces. 
distances in these spaces, and their relationship with image recognition are explained. 

1. Images 

An image consists of a set of pixels. For example, a 480x640 image will have 307.200 pixels. A 
pixel can have a set of values. For example, a pixel in a red color image can have three values: 
RGB = (Red, Green. Blue) = (255. 0. 0\ A grey image can reduce three numbers into a single 
number. For example, a grey pixel has a value = 128. For simplicity, in the rest of this section, 
we assume each pixel has a single value. 

There are two ways to describe an image. Each description yields different features of the image. 
We will first introduce these descriptions: we will then explain the image spaces and the 
distances between images in these image spaces. 

The first way to describe an image is to use a vector. For example, a 5 -pixel image is described 
as (10, 20. 30. 40. 50). A vector is also viewed as a point in a vector space: therefore an image 
can be viewed as a point in a vector space. For example, a 480x640 image can be viewed as a 
point in a vector space of 307.200 dimensions. 
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The second way to de scribe an image is to use a set of particles. First of all, a binary image 
(black-and-white) is de scribed bv a set. For example, a 5-pixel image is described as 10101. Let 
the pixels be described bv five elemen ts in a set: 0. 1. 2. 3. and 4. then the binary image 10101 
represents the set (0. 2. 41. Here is another example. 11100 represents the set (0. 1. 2k N 
elements will yield 2 N subsets. N pixels will yield 2 N black-and-white images. All of the 2 N 
subsets together form a space. A subset is a point in this space. All of the 2 N black-and-white 
images together form a space. A black-and-white image is a point in this space. Now we extend 
this definition of ima ges beyond the binary values. Consider each element in a set is a mass 
particle th at has an integer mass value, then a color image can be described as a set of mass 
particles. For example. (1 28. 0. 256. 0. 128) is a set of mass particles (0. 2. 4\ where particle 0 
has mass 128. particle 2 has mass 256. and particle 4 has mass 128. 

To summarize, an im age is a point in a space. The space can be a vector space, which consists of 
a set of vectors. The space can be a subset space, which consists of a set of subsets. 

In either description, the distance between two points can be defined, i.e. the distance between 

two images can be defined. In a vector space, the distances are L r -distances, where p = 1. 2 

oo. For simplicity, consider a two-dimensional vector space, then the distances are: 

k= \*z=m\ + I Yl-yil 
L9 = V(x9-x0 2 + (v,-v,) 2 

L m = max ( lx?-xi I . I y 7 -v. I I 
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In a subset space, the distance between two points is called the Hausdorff distance. We will only 
use an example to illustrate what the Hausdorff distance is. Consider two cities as two sets, the 
Hausdorff distance between the two cities is this: you can start from any point in one city, after 
traveling the Hausdorff distance toward the second city, you will be guaranteed to be in the 
second city. 

2. Mapping 

Mapping establishes a relationship between a Domain to a Range. Both a Domain and a Range 
are sets. For example, a Domain can be (a, b) and a Range can be (0, 11, then a mapping is (aO, 
bll. If both Domain and Range are a set of finite elements, when the Domain is removed, the 
Range will form a vector. In the above example, when the domain is removed. The Range is a 
vector, (0, 1). 

3, Markov Chain and Pattern Classification 

A Markov chain is a sequence of random values whose probabilities at a time step depends upon 
the value of the number at the previous time step. The controlling factor in a Markov chain is the 
transition-probability; it is a conditional probability for the system to go to a particular new state, 
given the current state of the system. The key feature of a Markov chain is its invariant 
distribution: after a Markov chain evolves long enough, the probability for each configuration to 
appear is fixed. 

For example, let a Markov chain have two states, 0 and 1 . The transition matrix elements are, tnn 
= t_ m = 0.5, tjm = 0, t_n = 1 . A possible evolution is 0001 11111... Another possible evolution is 
00000001 1 1 1 1 1 1 1 . The invariant distribution is f(0) = 0, f(\) = 1 . 



11 



This invariant distribution can be used to classify patterns as follows. Let x be an image, and let 
a. b be two classes. We can construct the two possible vectors: (x, a) and (x, b). Let an invariant 
distribution function of a Markov chain be z = F (v), where v is a vector. Assume v = (x, a), z = 
zl; and v = (x, b), z = z2, then this invariant distribution function actually classifies the image, x, 
as follows: the probability of x in class a is zl: and the probability of x in class b is z2. If we can 
further assume zl » z2, then x is classified as a member in class a. 

In particular, assume a Markov chain is constructed with an image p. Let x be an image, and let 0 
(No Match), 1 (Match) be two classes: then the two possible vectors are (x, 0) and (x, 1). After 
the Markov chain evolves long enough, an invariant distribution of the Markov chain is reached. 
Let a distribution function be z = F (v), where y is a vector. If y = (x, 0), z = zO; and y = (x, 1), z 
= zl, then the probability of x in class 0 is zQ and the probability of x in class 1 is zl. The 
matching score between image p and image x will be zl . 

A triplet is (x, 0 zO) or (x, 1, zl). A doublet is (x, zl). 

4. Markov Chain and Neural Network 

A particular type of neural network is called a Boltzmann Machine, in which each neuron has a 
certain probability to be in state 0 and a certain probability to be in state 1. A Boltzmann 
Machine forms a Markov chain. 

5. Artificial Neural Net 

The artificial neural net is a dynamical system, which will transit from one state to the next 
according to its internal connections. 
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The configuration s pace is set of all possible configurations. For example, assume there are 

three binary neurons and the set of all possible configurations is (000. 001. 010, 011 1111. 

Each configuration in this space is considered as a point. In general if there are N neurons, there 
will be 2 N points in the configuration space. 

An input i mage will ultimately be mapped into a neural net. The number of pixels, in general, 
does not agree with the number of neurons. Mapping an input image into a given neural net will 
be necessary. 

The connection space in this invention is the set of all possible connections. For example, 
assume there are three binary neurons I 0. 1, 21 and the set of all possible connections is 
{nothing, 0, 1, 2. 01. 02. 12, 012 1. where (0, 1. 21 are self connections. (PL 02. 121 are regular 
connections, and (0121 is a high order connection. Each connection in this space is considered as 
a point. In general, if there are N neurons, there will be 2 N points in the connection space. 
The synaptic connection matrix is a mapping from the connection space to real numbers. For 
example, the action of a neuron can be: 

= f(l. ax a*Xi ayirXi Xj X,, ... ) 

Where X i^ are neuron states (grounded or excited) at a time step. XN are neuron states at the next 
time step, ay aik a^ a^k ... are connection matrix elements, and f ( xl, x2, x3. are a function 
which governs the neuron transition. The points in the connection space is represented as 
follows: assume there are four binary neurons ( 0. 1. 2. 31 and the set of all possible connections 
is ( nothing; 0, 1, 2. 3: 01. 02. 03. 12. 13. 23: 012. 013. 023. 123: 01231. where (0. 1. 2. 31 are 
self connections. (01. 02, 03, 12, 13. 231 are regular connections, and (012. 013. 023. 123: 
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0123} are the high order connections. These connections are: 0000 = nothing, 1000 = neuron O's 

self-connection. 0100 = neuron Vs self-connection 1 100 = connection between neuron 0 and 

1 i-e- the connections are (0000. 1000, 0100, 0010. 0001. 110Q, 1Q1Q. iqqi, pup, qiqi. 

0011. 1110. 1101. 1011.0111. 1111} 

An input vector for a neural network or an input vector is a vector built from an image pixel 
array. The number of pixels, in general, does not agree with the number of neurons. Mapping an 
input image into a giv en neural net will be necessary. For example, let an 8-pixel grev image 
have the pixel array (10. 2 0. 30. 40. 50. 60. 70. 80^ and let a 4-neuron net be (0. 1. 2. 3k the 
input vector can be obtained bv pixel average: (15, 35. 55. 75V 
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SUMMARY OF THE INVENTION 

The present invention is different from Informix database where images can be queried by aspect 
ratio, brightness, global colour, local colour, shape, and texture attributes. The present invention 
is different from Imatch where images can be queried by colour similarity, colour and shape 
(Quick) , colour and ohapo (Fuzzy) , colour percentage, and colour distribution. The present 
invention is different from the Oracle8/ Enterprise Server where images can be queried by color, 
local color, shape, and texture attributes. 

The present invention is unique in its sample image, control proc e ss, control parameters, and 
learning and matching algorithms. The current algorithms do not use methodologies deployed in 
the above systems. In particular, the following parameters are not used: aspect ratio, brightness, 
global colour, local colour, shape, colour similarity, colour and shape (Quick) , colour and shape 
(Fuzzy) , colour percentage, and colour distribution, local color, shape, and texture attributes. The 
present invention has nothing in common with any existing system. 

Even though the current invention is applied to images, the algorithms in the invention can be 
applied to other types of data, such as sound, movie, . . . 

1. Process 

The present invention is a content-based image retrieval/recognition system, where users specify 
an image(s) or segment(s); adjust control parameters of the system, and query for all matching 
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images from an image directory or database. The user operation is generally divided into two 
phases: learning phase and search/recognition phase. In th e l e arning phas e , various typ e s of 
process e s, such as image pr e proc e ssing, imag e siz e r e duction, and imag e filtering ar e applied to 
the images. Then th e imag e s are sent send to a r e cognition module to toach th e modul e th e 
characteristics of th e imago as sp e cifi e d by an array of pix e ls. Each pix e l is d e fin e d by an 
int e ger, which can hav e any numb e r of bits. Th e l e arning modul e can use ABM or APN l e arning 
algorithms to l e arn th e sample imag e . Both the algorithms will be list e d in th e pres e nt inv e ntion. 
In th e s e arch/retrieval phas e , th e recognition modul e d e cid e s th e classification of an image in a 
s e arch directory or a s e arch database. 

The process roughly divided into several modules: processing, learning and recognition. Before 
learning and recognition, various types of processes, such as image preprocessing, and image 
filtering are applied to the images. 

In the learning phase, some sample images are sent to a learning module to teach the module the 
characteristics of the image as specified by an array of pixels. Each pixel is defined by an 
integer(s), which can have any number of bits. The learning module can use ABM or APN 
learning algorithms to learn the sample image. Both the ABM and APN learning algorithms are 
new inventions. 

In the search/retrieval phase, the recognition module searches a source such as all image in a 
folder, or all image in a database table. The recognition module can use ABM or APN 



16 



recognition algorithms to match the sample imagefs) against all images from a source. Both the 
ABM and APN recognition algorithms are new inventions. 

In a retrieval/recognition system, a "training" for the system or "learning" by the system is to 
teach the system what characteristics of an image, or a segment of an image (key) to look for. A 
system operator completes this step by specifying the sample image(s); specifying the 
parameters, and clicking on e button, the "training" button, which app e ars in th e graphical us e r 
int e rface of th e system and starting the "training" process by clicking a button . A "retraining" by 
the system is to teach the system what characteristics of images to look for with additional 
sample images , after the system is already trained. Training and retraining together allows the 
system to learn from many sample image(s) and segment(s) simultaneously. 

A "search" or "retrieval" is to look for matching images from an image source such as, directory, 
many directories, subdirectories, network, Internet, or database, etc. A system operator 
completes this step by specifying the image source such as search directory(s), specifying the 
parameters and clicking on e button, th e "s e arching" button starting "searching" process by 
clicking a button, which appears in the graphical user interface of the system. The results can be 
displayed within the software systems or displayed in a program created by the system. Twe 
particular applications ar e image v e rification (1:1 matching, binary output: y e s/no) and imag e 
id e ntification (1 :N matching, singl e output to indicat e a classification). 

Several applications for the invented ABM learning algorithm, APN learning algorithm, ABM 
recognition algorithm, and APN recognition algorithm are listed below: 
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Verification 

Verification is a one-to-one QjJJ Matching of a single sample image set against another. 
Generally, the first sample is newly captured and the second is the enrolled identifier on 
file for a particular subject. For example, in a user authentication environment, a score 
exceeding the threshold would return a 'match', resulting in the authentication of the user. 
A score below the threshold would return a 'no-match', resulting in the denial of access. 

Identification 

Identification is a one-to-many (1:N) Matching of a single sample image set against a 
database of samples, with no declared identity required. The single image is generally the 
newly captured sample and the database contains all previously enrolled samples. Scores 
are generated for each comparison, and an algorithm is used to determine the matching 
record, if any. Generally, the highest score exceeding the threshold results in 
Identification. 

Search/ Retrieval 

Search is similar to Identification, i.e. one-to-many (I :N) Matching; however, the result is 
a set of possible matching images, not a classification. Identification returns a 
classification, while Search returns multiple matched images. 

Classification 
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In the above three cases: only one class of imaged is compared with a set of images to 
be searched. " Classification" is manv-to-one (N:\) or manv-to-many QjjN) Matching. It 
specifies one i mage in one of many categories. Classification is a many-to-many (N: N) 
matching with a single output to indicate a classification. 

A "classification" or "r e cognition" is to rep e at training and s e arch for each category of imag e s. 
At the end, a system operator clicks one button, the "classification" button, which appears in the 
graphical us e r interfac e of th e syst e m. The results can b e display e d within the softwar e systems 
or displayed in a program created by the system. Classification is an N: N matching with a single 
output to indicate a classification. 

The param e ters and s e ttings of a particular operation can b e sav e d and recalled later. Clicking a 
button, cut and past e , op e n fil e s, or typing can bo achieved by r e calling can achi e v e r e calling a 
saved operation. The saved results are called "batch code". The "Batch" buttons provide means 
to ex e cut e th e s e sav e d batch codes. 

A "process" is a sequence of training and searching, or a classification, or a specification of a 
batch cod e and execution of a batch cod e . Th e y ar e further divid e d into a s e arch proc e ss, a 
classification proc e ss, and a batch proc e ss. 

After the operator completes a matching process, the results consists of a list of pairs; the pairs 
consist of the matched image and the "weight", which reflects how closely the selected image 
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matches the sample image(s). This list can b e sort e d or unsort e d. This list provides th e link to th e 
match e d imag e s so th e match images can b e vi e wed with a single click. 

"Syst e m integration" is to combin e a softwar e compon e nt, which is an impl e m e ntation of this 
inv e ntion, with an application interfac e . 

The A search process, which is applicable to retrieval, verification, and identification, is: 

1 . Enter key image into the system; 

2. Set training parameters and click the training button to teach the system what to look for; 

3. Enter search-directory(s); 

4. Set search parameter(s), and click the search button; 

5. The system output is a list of names and weights: 

• The weight of an image is related to the characteristics you are looking for (the weight is 
similar to an Internet search engine weight); 

• Click the name of each image and an image will pop up on the screen. 

Figure 1 is the flow chart version of this algorithm. 

The A classification process is: 

1 . Enter key image into the system for each class : 

2. Set training parameters and click the training Training button to teach the system what to 
look for; 
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3 . Enter search-directory (s) ; 

4. Set search parameter(s), and click the s e arch Search button; 

5. Repeat the above process for each class and then click the? "Record" button. At tho ond, click 
Click the "Classification" button. The output web page will first list the sample images for 
each class. Then it will list: 

• An image link for each image in the search directory; 

• The classification weights of this image in each search; and 

• The classification of this image as a link. 

A classific ation process consists of several Search processes. Each class has a key image. This 
key image is used to match against all images in a search-directory. The results consist of a set of 
matched images and their matching scores. An image in the search-directory can be matched 
with several key images . The highest weight determines the final classification of an image in the 
search-directory. Figure 7 is the flow chart version of this algorithm. 

A batch process consists of executing a batch code. Each search process or a classification 
process has its own batch code. A batch code for a search process is a text file that consists of: 

1. Key image to train the system: 

2. Training parameters: 

3. Search-directory to specify a set of images to be searched: 

4. Search parameters. 
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A batch code for a clas sification process is a text file, that consists of several sets of parameters, 
one set for each class. Each set of parameters consist of: 



1. Key image to train the system: 

2. Training parameters: 

3. Search-directory to specify a set images to be searched: 

4. Search parameters. 

The batch code is gen erated automatically: for example, click a Save button to save the current 
setting, including kevf s). search directory(s). and parameters, into a batch code. The batch code 
can be vie wed bv clicking a button to recall one of the many batch codes saved earlier. The batch 
execution can duplicate the Search Process and the Classification Process in two clicks in a 
software i mplementation: the first click opens the proper batch code that needs to be executed 
and the second click runs the batch code. 

The batch process is: 

1. Provide the batch code to the system, which includes: 

• Click the save Save button to save the current setting, including key(s), search 
directory(s), and parameters into a batch code. 

• Click a file File button to recall one of the many batch codes saved earlier. 

• Cut and paste or simply type in a batch code by keyboard. 

2. Click feateh Batch button to execute the code. 
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Figure 8 is the flow chart of the batch process. 

An int e gration proc e ss is to combin e a softwar e compon e nt, which is an impl e mentation of this 
inv e ntion, with an application int e rfac e . This invention also sp e cifi e s a us e r graphical int e rfac e 
for th e int e gration. 

2. Parameters 

The search, classification, and batch processes require a set of parameters. All the parameters can 
be specified in the a syst e m user interface , e ither through clicking buttons or through Windows . 
The parameters are specially related to the ABM and APN algorithms , which will b e claim e d in 
this pat e nt . Figure 3 and 4 show a sample implementation of these parameters. 

The "Area of Interest" specifies an image segment, which is specified by 4 numbers: the 
coordinates of the upper-left corner and the bottom-right corner. 

The "internal representation" specifies the dimensions of a pixel array used for computation, 
which may or may not be the actual image pixel array. 

The "Background" or "Background filter" selects an image-processing threshold filter th e pixel 
array must pass through b e for e e nt e ring th e l e arning compon e nt of th e syst e m . 
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The "Symmetry" represents similarity under certain types of changes, such as int e nsity 
Intensity , translation Translation symmetry, Scaling, Rotation, oblique Oblique, combined 
rotation Rotation and scaling Scaling or any combination thereof. 

The "Rotation Types" specify the range of rotation if the rotation symmetry is used. Examples 
are 360°-rotations, -5° to 5° rotations, and -10° to 10° rotations, or other settings that fit the 
user's need. 

Th e "Reduction Typo" sp e cifies th e method used when r e ducing a larg e imag e pixel array to a 
smaller pix e l array. 

The "Sensitivity" deals with the sample segment size; high sensitivity is for small segment(s) and 
low sensitivity is for large segment(s). 

The "Blurring" measures the distortion due to data compression, translation, rotation, scaling, 
intensity change, and image format conversion. 

Th e "Shap e Cut" is to e liminat e many imag e s that have diff e r e nt shap e s from the sample 
s e gm e nt. 

Th e "Ext e rnal Weight Cut 1 ' is to list only those retriev e d imag e s with weights gr e at e r than a 
certain value. The weight Cut is an integer gr e at e r than or equal to 0. Th e re is no limit how largo 
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this int e g e r can bo. The "Internal Weight" Cut plays a similar rol e as th e Ext e rnal Cut in a 
percent valu e rather than an absolut e weight valu e . 



Th e "Imag e Typ e " sp e cifi e s the l e arning compon e nt wh e th e r to tr e at th e pix e l array as black and 
whit e imag e s or a color image. It also instructs th e l e arning component whether to use a 
maximum valu e , int e gration, or both. 

Th e "L/S S e gment" (Larg e /Small segm e nt) sp e cifi e s th e syst e m wh e r e to focus wh e n s e arching 
imag e s. 

Th e "Short/Long" s e arch sp e cifi e s an imag e sourc e s uch as wh e th e r to s e arch on e dir e ctory or 
many directori e s. 

Th e "Short Cut" is a Scrollbar to sel e ct an int e ger b e tw e en 0 and 99; e ach integer is mapp e d to a 
set of pr e d e fin e d settings for th e param e ters. 

Th e "Bord e r Cut" controls th e portions of imag e s to b e us e d in th e imag e r e cognition. 

Th e "S e gm e nt Cut" controls th e thr e shold us e d to reduce an image into an int e rnal 
repr e s e ntation. 



3. System Layout 
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Attrasoft Component-Object structure consists of three layers (See Figure 2): 



• Application Layer 

• Presentation Layer 

• ABM Network Layer 

The ABM Network Layer has two algorithms to be claimed in the present invention: 

• ABM (Attrasoft Boltzmann Machine); 

• Attrasoft PolyNet (APN): multi-valued ABM. 

This layer is responsible for learning and classification. 

The Presentation Layer is an interface between the ABM net layer and the user interface layer. 
There are two types of data used by the systems: user data or application data, and ABM neural 
data. ABM networks use ABM neural data. User data depends on the application. The 
presentation layer converts the image data into neural data used by the ABM layer component. 

The Application Layer is the front-end graphical user interface, which the users see directly. This 
layer collects all parameters required for necessary computation. 

4. Algorithms 
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The ABM layer deploys two algorithms, ABM and APN. The ABM and APN algorithms consist 
of a combination of Markov Chain Theory and the Neural Network theory. Both theories are 
well known. The ABM and APN algorithms are newly invented algorithms, which have never 
been published. 

The following terms are well known: Markov chain, state of Markov chain, invariant 
distribution. 

The basic flow chart for ABM and APN algorithms are: 

1 . Combine an image and its classification into a vector. 

2. All such together form a mathematical configuration space. Each point in such a space is 
called a state. 

3. A Markov chain exists in such a space where the state of the configuration space is a state 
of the Markov chain. 

4. The construction of such a Markov chain is by a particular type of neural network, called 
ABM network or APN network. This type of neural net satisfies 3 features: (1) fully 
connected; (2) the order of the neural net is the same as the number of neurons in the 
network, i.e. the number of connections is an exponential function of the number of 
neurons; and (3) the connections follow particular algorithms, known as ABM and APN 
algorithms. 
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5. The Markov chain will settle on its invariant distribution. A distribution function is 
deployed to describe such a distribution. In particular, such distribution function 
classifies the images. 

4. Th e Markov chain will s e ttl e on its invariant distribution. A distribution function is 
d e ploy e d to d e scribe such a distribution. In particular, such distribution function 
classifies the images. 

5. Th e construction of such a Markov chain is by a particular typ e of n e ural n e twork, call e d 
ABM network or APN network. This typ o of neural not satisfies 3 features: (1) fully 
connect e d; (2) th e ord e r of th e n e ural n e t is th e sam e as the number of n e urons in th e 
n e twork, i.e. th e number of connections is an exponential function of the number of 
n e urons; and (3) th e conn e ctions follow particular algorithms, known as ABM and APN 
algorithms . 

Explanation: 

In Step L the explanation on how to combine an image and its classification into a vector 
is given in the TECHNICAL BACKGROUND section. 

In Step 2. no particular action will need to be taken. This is a conceptual step, which 
views an image as a point in a space or several spaces, resulting in the definition of 
distances between points. The distance is further used in various parameters and 
matching computations. The explanation on how to treat an image as a point in an image 
space is given in the TECHNICAL BACKGROUND section. 
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In Step 3, a Markov chain is formed. Again, no particular action will need to be taken. 
This is a conceptual step, which specifies how the image matching will be implemented 
via the invariant distribution of a Markov chain. 

In Step 4, an artificial Neural Network is formed to implement the Markov chain. Both 
the ABM and the APN satisfies 3 features: 

(1) Fully connected: 

(2) The order of the neural net is the same as the number of neurons in the network, 
i.e. the number of connections is an exponential function of the number of 
neurons: and 

(3) The connections follow particular algorithms, known as the ABM and the APN 
algorithms. 

Figure 10 gives an example of a fully connected artificial neural network with 4 neurons. 
Figure 1 1 gives an example of a Markov chain generated by a neural net with 4 neurons 
in Figure 10. The controlling factor in a Markov chain is the transition-probability 
Matrix. Assume the neurons are 13, 2, LP} in Figure 10. Each neural state is a state in 
the Markov chain. These states are: 0000 = 0: 0001 ~ 1: 0010 = 2: ...: 1111 = 15. The 
states are 0000 = 0: 0001 = 1: 0010 = 2: ...: 1 1 1 1 = 15, i.e. the Markov chain state 0 is a 
neuron state where all neurons are grounded: the Markov chain state 1 is a neuron state 
where all neurons are grounded except neuron 0: the Markov chain state 2 is a neuron 



29 



state where all neurons are grounded except neuron 1; the Markov chain state 3 is a 
neuron state where neurons 2 and 3 are grounded and neurons 0 and 1 are excited; ... 

In Step 5. the Markov chain will settle on its invariant distribution, which can be used for 
identification. Let x be an image, and assume there are two classes: class 0 stands for 
'No Match' and class 1 stands for 'Match'; then the two possible vectors are (x, 0) and 
(x, 1). Let an invariant distribution function of the Markov chain, constructed by the 
neural network, be z = F (y), where v is a vector. If v = (x, 0), z = zO; and y = (x, 1), z = 
zl, then the probability of x in class 0 is zO and the probability of x in class 1 is zl. The 
result will be {(x, 0, zO), (x, 1, zl)}. The (x, 1, zl) will be the computation result. For 
simplicity, (x, 1, zl) is further simplified to (x, zl). The users will see results (x, zl) 
directly in the output of the system. Figure 5 shows a possible implementation of the 
Search process results and Figure 6 shows a possible implementation of the Classification 
process results. 

Th e Step 4 of th e abov e is d e fin e d as follows: 

L e t x b e an imag e , and l e t a, b b e two class e s; th e n th e two possibl e vectors ar e (x, a) and (x, b). 
L e t a distribution function b e z = F (y), wh e re y is a v e ctor. If y - (x, a), z = zl; and y = (x, b), z 
~ z2, th e n th e probability of x in class a is zl and th e probability of x in class b is z2. Th e r e sult 
will b e ((x, a, zl), (x, b, z2)}. Th e us e rs will s ee r e sults like this dir e ctly in th e output of th e 
syst e m. 
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In the ABM or APN algorithms, content-based image retrieval and image recognition are 
basically the same problem; therefore, they can be converted from one to the other. To convert 
from an image search problem to an image recognition problem, one query is required for each 
class. To see whether an image, say B, is in class A, you first train ABM with all images in class 
A, then try to retrieve image B. If image B is not retrieved, then image B is not in class A. If 
image B is retrieved only for class A, then image B is in class A. If image B is retrieved for 
several classes, the class with the largest relative probability is the one to which image B 
belongs. Image search is an image classification problem with only 1 class. 

ABM is a binary network. APN is a multi-valued network. 

5. Components and Application-Programming Interface 

Software components can be isolated to be attached to different front-end systems. Figure 2 
shows the three-layer architecture of this invention. This can be done with ABM neural layer 
alone, or both ABM layer and presentation layer. The ABM layer component is a core of the 
present invention. The value of such a sub-system is the same as the whole system. 

This inv e ntion also d e fines th e application programming int e rfac e (API), which sp e cifi e s th e 
syst e m integration. This API is call e d IVI API. 



31 



BRIEF DESCRIPTION OF VIEWS OF THE DRAWING 

Figure 1 shows the algorithm of tho Search Procooo a n Image Retrieval Process, which is 
applicable for image verification, identification, and retrieval. 

Figure 2 shows a 3 -Layer Attrasoft Internal Architecture. 

Figure 3 shows a sample User Interface of the Present Invention. 

Figure 4 shows a sample Key Input for the Present Invention. 

Figure 5 shows a sample S e arch Retrieval Output of the Present Invention. The search output is a 
list of pairs. For example, line one of the Figure 5 shows a double (image, score), where the 
image is "D:\... GODWIN 1 A. JPG", and the score is 24192. 

Figure 6 shows a sample Classification output of the Present Invention. The classification output 
is a list of triplets. The N:N (many to many) classification is implemented by a set of 1:N (one to 
many) searches. Let a and b be two different classes. Given an image, x. we can perform a 1 :N 
(one to many) search for class a. with a triplet (x. a. score-x-a); and a 1:N (one to many) search 
for class b. with a triplet (x. b. score-x-b). The final answer is (x. score-x-a. score-x-b: 
maxlscore-x-a. score-x-M \ In the above figure, the first line, "class 1 9.jpg" is an image, x. The 
second line has 9 scores, indicating how likely the image x belongs to anv of the 9 classes. 
Finally. Class 0 is the best match because it has the maximum score. 193784. The training image 



32 



for class 0 is "D:/... /class l.jpg" The third line in the above figure shows the classification result 
of the image, "class 1 9.jpg". which is also a link to the training image. "D:/.../classl.jpg". 

Figure 7 shows the Classification Process, which consists of a multiple-search process in Figure 
L 

Figure 8 shows the Batch Process, which allows users to duplicate a Search or Classification in 
two clicks. 

Figure 9 shows the ABM and APN Algorithm Flow Chart. 

Figure 10 shows an example of a fully connected artificial neural network with 4 neurons. The 
controlling factor in a Markov chain is the transition-probability Matrix. Let pj be the transition 
probability matrix element from state i to state i. For example, pn is the transition probability 
from neuron state 0001. where the neuron 0 is excited and the rest are grounded, to neuron state 
0010. where the neuron 1 is excited and the rest are grounded. Each neural state is a state in the 
Markov chain. Assume the neurons are (3. 2. 1. 0}. The states are 0000 = 0; 0001 = 1: 0010 = 2: 
1111 = 15. For this Markov chain, state 0 is a neuron state where all neurons are grounded: 
state 1 is a neuron state where all neurons are grounded except neuron 0: state 2 is a neuron state 
where all neurons are grounded except neuron 1; state 3 is a neuron state where neurons 3 and 4 
are grounded and neurons 0 and 1 are excited: . . . 
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Figure 1 1 shows the M arkov chain generated bv the neural net with 4 neurons in Figure 10. The 
controlling factor in a Markov chain is the transition-probability Matrix. Each neural state is a 
state in the Markov chain. The states are 0000 = 0; 0001 = 1: 0010 = 2: 1111 = 15. 

Figure 12 shows the More Detailed ABM Algorithm Flow Chart. 

Figure 13 s hows the More Detailed APN Algorithm Flow Chart. The APN algorithm is an 
extension of the ABM algorithm. The ABM algorithm is for binary images and the APN 
algorithm is for multi-valued images. The APN algorithm will first use the ABM algorithm to 
make an i mage matching for binary images. If there is no match from the ABM algorithm, the 
ABM algorithm will output a score 0. which will remain to be 0 when the APN algorithm 
finishes. If there is a match for the ABM algorithm, then the APN will further modify the ABM 
results bv comparing two input vectors generated bv the two comparing images. For example, 
consider two images (3 5 0 0) and (1 2 0 01 their binary versions are 1 100 and 1 100. The ABM 
algorithm will compare the binary images. 0011 and 1100. and return a match. The APN 
algorithm will take the ABM results, and modify it bv the distance between (3 5) and (\ 2). 

Figure 14 shows the Connection Space and the Sensitivity distance. (a) Three-dimensional 
connection space, (b) Four-dimensional connection space. Assume two connections are 1 100 and 
1111. then the Sensitivity distance between them is 2. if the L i distance is used, where I4 = | x? 
- xi I + I v? - vx I + ... Here 1100 means that neuron 0 and neuron 1 have a non-zero 
connection matrix element; and 1111 means there exists a non-zero matrix element between 
neuron 0. 1. 2. and 3. 
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Figure 15 shows the Image Space and the Blurring distance, (a) Three-dimensional binary image 
space, (b) Four-dimensional binary image space. Assume a binary image is 1 1 1 L we can draw a 
"sphere" around 1111 with a radius of 1 . The following binary images are in this sphere: 1111. 
0111.1011.1101. 1110. All these images in this "sphere" will be considered the same as the 
original image. 1111. Blurring is a user-defined parameter that defines this radius. Any image in 
this sphere is as good as the original image. 1 1 1 1. for image matching. 

Figure 16 shows the ABM Learning Algorithm Flow Chart. 

Figure 17 shows APN Learning algorithm, which is an extension of the ABM algorithm for non- 
binary values. For example, consider two images (3 5 0 0) and (1 2 0 0). their binary versions are 
1 100 and 1 100. The ABM algorithm will compare the binary images. 1 100 and 1 100. and return 
a match. The APN training algorithm will first train the binary ABM net, and then store an 
additional mapping (0"» 3. !■» 5) for each connection. 

Figure 18 shows the ABM Recognition Algorithm Flow Chart. 
Figure 19 shows the APN Recognition Algorithm Flow Chart. 
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DETAILED DESCRIPTION OF THE DISCLOSED EMBODYMENT 



Preferred Embodiment of the Search System 

An image search/classification constructed in accordance with the preferred embodiment 
comprises a computer-based workstation including monitor, keyboard and mouse, a content- 
based image retrieval software system and a source of images. 

The source of the images may be on the local drive, network or the Internet. The source is 
connected to the workstation. The source of images may be accessed directly via open files, or 
indirectly, such as going into a file to find the images or going into a database application to find 
the images, etc. 

The preferred workstation can be a PC or any other type of computer comput e rs , which connects 
to a data source. 

The preferred content-based image retrieval software system is any software, which has ABM or 
APN algorithm as a component. It can be a Window-based system, or any other operating system 
based systems, or Internet based systems. 

About This Invention 

This invention consists of main newly invented processes: 
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• ABM Process 

• APN Process 

Within each process, there are two newly invented algorithms. For the ABM Process, the new 
inventions are: 

• ABM Learning Algorithm; 

• ABM Recognition Algorithm. 

For the APN Process, the new inventions are: 

• APN Learning Algorithm; 

• APN Recognition Algorithm. 

Overview of the ABM Algorithm Process 

The following terms are well known: synaptic connection or connection. 

The basic flow chart for ABM algorithm process is: 

1 . Create an ABM net with no connections; 

2. Combine an image and its classification into an input vector. 

3. Impose the input vector to th e l e arning modul e ABM Net . 
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4. The ABM neural connections are calculated based on the input vector. Let N be the 
number of neurons; the order of connections can be up to N and the number of 
connections can be 2**N, where ** represents the exponential function. 

5. The Markov chain is formed after the connections are established. This Markov chain 
will settle on its invariant distribution. A distribution function is deployed to describe 
such a distribution. 

6. This distribution function, once obtained, can be used to classify images. This will 
produce triplets of ( image, class, and weight ). Image retrieval and classification are two 
different sides of the same token. 

7. These triplets of image, class, and weight can be viewed as the results of the 
classification process. For the retrieval/ search process, a doublet of image and weight are 
displayed. The second part of the triple is omitted because the search problem has only 
one class. 

Explanation: 

Step 1. An empty neural net is created. It looks like the graph in Figure 10 without any 
connections. 

Step 2. The explanation on how to combine an image and its classification into a vector is 
given in the TECHNICAL BACKGROUND section. 

Step 3. The ABM net will read an input vector. The Presentation Layer in Figure 2 
prepares the input vector. In general, the image size and the ABM neural net size do not 
match: for example, an image is 480x640 and a neural net is 100x100. The Presentation 



38 



Layer in Figure 2 will reduce the image pixel array to fit the ABM neural network 
through various methods. This procedure will be specified in more detail later in the 
"Presentation layer ..." section. 

Step 4. The ABM will be trained. This step will be further expanded later in the "ABM 
Training Algorithm" section. 

Step 5. The ABM net generates a Markov chain. The ABM net is a fully connected neural 
net following the rule of the Boltzmann Machine, i.e. each neuron has a certain 
probability to be excited and a certain probability to be grounded. Such a neural net is a 
Markov chain. Figures 10 and 11 show the relationship between a neural net and a 
Markov chain. No particular action will need to be taken in this step. This is a conceptual 
step, which specifies how the image matching will be implemented via the invariant 
distribution of a Markov chain. 

Step 6. An invariant distribution of the Markov chain is reached, which can be used for 
identification. This step will be further expanded later in the "ABM Recognition 
Algorithm" section. 

Step 7. The results are presented. 

Figure 12 shows the ABM Process Flow Chart. ABM is for black-and-white images and APN is 
for color images. The basic approach, including the base Markov chain, is identical. The only 
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difference is that the APN will construct two extra vectors, one from the training image and one 
from each search image. These two vectors will be used to modify the scores of the ABM 
Process, i.e. the invariant distribution of the Markov chain. 

Overview of the APN Algorithm Process 

The APN Process is an extension of the ABM Process. The ABM Process is for binary images 
and the APN Process is for multi-valued images. The APN Process will first use the ABM 
Process to make an image matching for binary images. If there is no match from the ABM 
Process, the ABM Recognition algorithm will output a score 0. which will remain to be 0 when 
the APN Recognition algorithm finishes. If there is a match for the ABM Recognition algorithm, 
then the APN will further modify the ABM results by comparing two vectors generated by 
comparing two images. For example, consider two images (3 5 0 0) and fl 2 0 0\ their binary 
versions are 1100 and 1100. The ABM Recognition algorithm will compare the binary image. 
1 100 and 1 100. and return a match. The APN Recognition algorithm will take the ABM results, 
and modify it by comparing (3 5) and (1 2). 

The basic flow chart for APN algorithm Process is: 

1 . Create an APN neural net with no connections; 

2. Combine an image and its classification into an input vector. 

3 . Impose the input vector to th e learning modulo APN Net . 
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4. The APN neural connections are calculated based on the input vector. Let N be the 
number of neurons; the order of connections can be up to N and the number of 
connections can be 2**N, where ** represents the exponential function. 

5. A mapping over each connection is established. Let K be a number of neurons in a K 
order connection, where K is less than or equal to N, then there this will be a K to K 
mapping, i.e. the domain of the mapping has K integers and the range of the mapping has 
K integers. 

6. The K-elements mapping is changed to N-element mapping by adding (N - K) pairs of 0 
to 0 relations for each of the neurons not in the set K. By taking the domain of this 
mapping away, the range of this mapping forms a vector, APN connection vector. 

7. The Markov chain is formed after the connections are established. This chain will settle 
on its on its invariant distribution. A distribution function is deployed to describe such a 
distribution. 

8. This distribution function, once obtained, can be used to classify images. This will 
produce triplets of image, class, and weight. 

9. Comparing the input-vector and the APN-connection-vector modifies this weight. This 
will produce a new set of triplets of image, classification, and weight. 

10. These triplets of ( image, class, and weight } can be viewed as the results of the 
classification process. For the search process, a doublet of image and weight are 
displayed. The second part of the triple is omitted because the search problem has only 
one class. 

Explanation: 
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Step L see Step 1 of the ABM net above. 
Step 2, see Step 2 of the ABM net above. 
Step 3, see Step 3 of the ABM net above. 

In Step 4, the underlying ABM will be trained. See Step 4 of the ABM net above. 
Step 5 represents the extensions from a binary ABM net to a multi-value APN net. In the 
TECHNICAL BACKGROUND section, we have stated the black-and-white image is a 
subset and the color image is a subset of particles with mass. The K-to-K mapping in this 
step represents the mapping from a particle to its mass; therefore, the images used are not 
black-and-white images, but color images. For a given training image, this mapping is 
defined by the input image. For example, let N = 4 and K = 3, and let a training image be (10, 
20, 30, 0\ then the base black-and-white image is 1 1 10 and the mapping is (0 -> 10, 1 ->20, 
and 2->301. For both the training image and search image, this K-to-K mapping will be 
computed in a similar way. 

In Step 6, the K-to-K mapping will be modified twice. It will first be converted into N-to-N 
(many to many) mapping first. In the above example, this new mapping is 10 -» 10, 1 ->20, 
2->30, 3 -> 01. Then it will be normalized. For example, the new mapping is the old 
mapping divided by 2: 10 ± 5, 1 -^10, 2->15, 3 ± 01. Now the Domain of the mapping, 
10, 1, 2, 3} is removed and the Range of the mapping converted into a vector (5, 10, 15, 01. 
This APN connection vector is the difference between the ABM net and the APN net. 
Step 7, see Step 5 of the ABM net above. 
Step 8, see Step 6 of the ABM net above. 

Step 9, the results in Step 8 are modified in this step. The distance (L t -distances, where p = 
L_2 oo) between the new APN connection vector generated by an input search image and 
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the APN connection vector generated by the training vector is computed. This distance 
modifies the results in Step 8. It could be as simple as dividing the old weights by the 
distance. 

Step 10, see Step 7 of the ABM net above. 
Figure 13 shows the ABM Process Flow Chart. 



User Interface Layer of software for implementation of ABM and APN 
Algorithms 

There are three major operations: 



• Search or rRetrieval; 

• Classification; and 

• Batch. 



These are the principle modes of the system that runs on the workstation. The software executed 
in these three modes can have various user interfaces, such as in Windows environment or the 
web environment, etc. The user interface collects necessary information for the computation. 



A Search process has two phases: learning and recognition. In the learning phase, a key image is 
used to train the proposed system what to look for by using the ABM or the APN learning 
algorithm. In the recognition phase, the proposed system searches through all images in a search- 
directory or a search-database for comparisons via the ABM or the APN recognition algorithm. 
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The sample outputs are given in Figure 5 and Figure 6. The User Interface Layer is responsible 
for collecting all parameters, including key images and sources of images. The User Interface 
Layer will also pass the parameters to the translation layer, which converts an image into an 
input vector. 

Oth e r than th e k e y and tho a source of images, th e us e r int e rfac e may or may not pass th e 
following information to the next lay e r: 

Th e "Area of Int e r e st" sp e cifies an imag e s e gm e nt by two clicks. Th e s e two clicks g e n e rat e 4 
numb e rs th e coordinat e s of th e upp e r l e ft corn e r and the bottom right corn e r. S ee a sam ple 
impl e mentation in Figur e 4 for th e coordinat e s of the upp e r l e ft corner and th e bottom -rightr 

Th e "int e rnal r e pr e s e ntation" sp e cifies th e dim e nsions of a pix e l array us e d for computation, 
which may or may not b e th e actual imag e pix e l array. 

Th e "Background" or "Background filt e r" s e lects an imag e proc e ssing filter th e pix e l array must 
pass through b e for e e nt e ring th e l e arning compon e nt of th e syst e m. Th e int e rfac e will b e 
r e sponsibl e for sel e cting on e of many availabl e filters. 

Th e "Symmetry" r e pr e s e nts similarity und e r c e rtain typ e s of chang e s, such as int e nsity, 
translation symm e try, Scaling, Rotation, obliqu e , combin e d rotation and scaling or any 
combination th e r e of. For th e translation symm e try, this is impl e m e nt e d by physically translating 
th e sampl e imag e to all possibl e positions. The similar m e thods can b e appli e d to other 
symm e tries. 
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Th e "Rotation Typ e s" sp e cify tho rango of rotation if tho rotation symmetry is used. Examples 
are 360° rotations, 5° to 5° rotations, and 10° to 10° rotations, or other settings that fit the 
us e r's n ee d. 

Th e "Reduction Typ e " sp e cifi e s tho method used wh e n r e ducing a large imag e pix e l array to a 
small e r pixel array. 

The examples of the parameters are various distance used by the algorithms, such as Sensitivity, 
Blurring, and Shape Cut. These are various distances calculated by the system in the image space 
and the connection space, which are introduced in the TECHNICAL BACKGROUND section. 
The users set up the thresholds for these distances and the system computes these distances 
between two images. If a distance is greater than the user-specified value, the images do not 
match. Within the user specified range, the smaller these distances are, the higher the matching 
scores will be. 

The "Sensitivity" deals with the sample segment size; high sensitivity is for small segment(s) and 
low sensitivity is for large segment(s). This is a method to limit the relevant neural connections. 
The connection space is given in the TECHNICAL BACKGROUND section. When the ABM 
net, xl, is trained, there will be certain connections. All possible connections together form a 
space, HL For the ABM net with N neurons, such a space will have a maximum of 2**N 
connections, where ** is the exponential function. Each trained ABM net will have a set hi, 
representing non-zero connections. When deciding whether an image, 12, in a search directory is 
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a match to the current sample image. II. this image 12 can be turned around to train the new but 
similar A BM neural net. x2. This will generate a set of connections. h2. Sensitivity determines a 
maximum distance in the Connection Space, d. between hi and h2. Given a search image, which 
generates h2. if the distance between hi and h2 is greater than the Sensitivity, then there is no 
match between the input search image and the key image. Users set Sensitivity as a parameter of 
the proposed system; see the Sensitivity button in a sample implementation in Figure 3. For 
example, assume there are four binary neurons (0. 1. 2. 31 and the set of all possible connections 
is ( nothing; 0. 1. 2. 3: 01. 02. 03. 12. 13. 23: 012. 013. 023. 123: 01231. where (0. L 2. 31 are 
self connections. {01. 02. 03. 12. 13. 23} are regular connections, and 1012. 013. 023. 123:01231 
are the high order connections. These numbers are the subscripts of a synaptic connection matrix 
element that defines the action of a neuron: 

Each connection in this space is considered as a point. Assume one image generates a set, hi = 
(OK 02. 03. 12. 13. 231. and a second image generates the same set. h2 = (01. 02. 03. 12. 13. 
231. Assume the I4 distance is used for Sensitivity distance, where I4 = I x? - x^ I ± I y? - Vj 1 
+ .... then the Sensitivity distance between hi and h2 is 0. Assume one image generates a set hi 
= (011 and a second image generates a set. h2 = (01231. then the Sensitivity distance between 
hi and h2 is 2. because the distance between 1 100 and 1111 is 2. The best way to compute the 
Sensitivity distance is to use the connection space in the TECHNICAL BACKGROUND section, 
which converts the connections into binary strings. For example. 01 ■» 1 100. 02 ■» 1010. 03 ■> 
1001. Figure 14 shows the connection space. 

The "Sensitivity" deals with the oampl o segment size; high sensitivity is for small s e gmont(s) and 
low sensitivity is for largo sogmont(o). This is a method to limit th e relevant n e ural connections. 
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Wh e n ABM n e t, xl, is trained, thero will b e c e rtain conn e ctions. All possibl e conn e ctions 
tog e th e r form a space, HI. For th e ABM n e t with N n e urons, such a spac e will hav e a maximum 
of 2**N point, where ** is th e e xponential function. Each trained ABM not will have a s e t hi', 
repr e s e nting non z e ro connections. When d e ciding wh e th e r an image, 12, in a search dir e ctory is 
a match to the current sample imago, II, this image 12 can b e turned around to train the new but 
similar ABM n e ural n e t, x2. Th e will g e n e rat e a s e t of conn e ctions, h2. Similarity d e t e rmines a 
maximum distanc e , d, e ither using th e Hausdorff distanc e or LI distanc e or L2 distanc e . In th e 
connection space, starting from th e conn e ction s e t, h2, of the new ABM n e t, aft e r applying this 
n e w distance, d, a now s e t, h3, is obtained. Obviously the smaller this distanc e , d, is, th e small e r 
this now sot, h3, wall bo. This now sot, h3, is th e n transformed back to hi. Any point in hi but 
not in h3 will bo consid e r e d "too far" and th e r e for e is set to 0 for th e curr e nt imag e , 12, in th e 
s e arch directory. This r e duction in th e conn e ctions spac e is det e rmin e d by the s e nsitivity. 

The "Blurring" measures the distortion due to data compression, translation, rotation, scaling, 
intensity change, or image format conversion. For example, when an image in the bitmap format 
is saved into the ipeg format, the pixel values changes slightly, which are not visible to human 
eves. An image is a point in an image space. The image space is given in the TECHNICAL 
BACKGROUND section. All possible images (points) together form a space, the image space. 
An image is a point in such a space. When deciding whether an image, 12, in a search directory is 
a match to the training image, II, the distance, d (II, 12) can be calculated. Blurring enlarges this 
comparison. In the image space, any image in I2's neighborhood is just as good as 12. Blurring 
determines the radius of this neighborhood. I2's neighborhood consists of all images with a 
certain distance from 12. The distance can be either the Hausdorff distance, or LI distance, or L2 
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distance (See the TECHNICAL BACKGROUND section), or multiple distances. The maximum 
radius to define the neighborhood is Blurring or Blurring distance, which is specified by the user 
as a parameter, i.e. users enter a value for this parameter in the user interface; see the Blurring 
button in a sample implementation in Figure 3. For example, let x =1 1 1 1 be a four-pixel binary 
image, the following images have a distance of 1 from the original image x: 0111. 101 L 1101. 
1110. Figure 15 shows the binary image space. A "sphere" with radius 1 around x will consist of 
a center x =11 11. and 01 11. 1011. 1101. 1110. 

Th e "Blurring" m e asur e s th e distortion due to data compr e ssion, translation, rotation, scaling, 
int e nsity chang e , and imag e format conversion. This m e thod e xpands an image in th e s e arch 
dir e ctory from a singl e point to a set as follows. All possibl e imag e s tog e th e r form a spac e , th e 
imag e spac e . An imag e is a point in such a spac e . Wh e n d e ciding whether an imag e , 12, in a 
s e arch dir e ctory is a match to the curr e nt sampl e imag e , II, this imag e 12 can b e turn e d a small 
s e t around th e 12. L e t th e set b e IS2. Blurring d e t e rmin e s a maximum distanc e , d, eith e r using th e 
Hausdorff distanc e or LI distanc e or L2 distanc e . In the image spac e , starting from th e 12, aft e r 
applying this n e w distance, d, a n e w sph e re s e t, IS2, is obtain e d. Obviously th e small e r this 
distanc e , d, is, the smaller this new s e t, IS2, will be. Now any point in this s e t, IS2, is just as 
good as 12. This e xpansion in the imag e spac e is det e rmin e d by th e Blurring. 

The "Shape Cut" is to eliminate many images that have different shapes from the sample 
segment. All possible images together form a space, the image space. The image space is given 
in the TECHNICAL BACKGROUND section. All possible images together form a space, the 
image space. An image is a point in such a space. There are several distances defined in this 
image space, including L P -distances, where p = 1. 2 oo and Hausdorff distance. When 
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deciding whether an image, 12. in a search directory is a match to the current sample image. II. 
the distance, d (II. 12) can be calculated, "Shape Cut" determines a maximum distance. D. either 
using the Hausdorff di stance. LI distance. L2 distance, or multiple distances. If the distance 
between II and 12 is gre ater than "Shape Cut", d (IK 12) > D. then there is no match between the 
input search image an d the kev image. A user specifies the Shape Cut as a parameter, i.e. users 
enter a value for this parameter in the user interface (Figure 3 and Figure 4). 

The "Shape Cut" is to eliminat e many images that have different shapes from th e oamplo 
s e gm e nt. All possible images tog e ther form a spac e , th e imag e spac e . An imag e is a point in such 
a space. When deciding whether an imag o , 12, in a search directory is a match to the current 
sample imago, II, the distance between II and 12, d, can be determined, either using the 
Hausdorff distance or LI distanc e or L2 distance. If this distance, d, is larger than a 
predetermined distanc e , D, a mismatch can bo declared without going through the ABM neural 
net. This predetermined distanc e , D, is sot by the "Shape Cut" parameter. 

Th e "External W e ight Cut" is to list only those retri e v e d images with weights gr e at e r than a 
c e rtain value. Tho weight Cut is an integer greater than or oqual to 0. Thoro is no limit as to -hew 
larg e this int e ger can bo. 

Tho "Internal Weight Cut" plays a similar role as tho "External Cut" in a percent valuo rath o r 
than an absolut e weight valuo. 
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The "Imago Typo" sp e cifies the ABM or APN algorithm. It also instructs the neural layer 
compon e nt how to comput e th e weights. Th e weight can be comput e d by using th e invariant 
function of th e Markov chain, or integration of all contributions in th e tim e evolution of th e 
Markov chain, with or without r e aching th e invariant distribution. 

The "L/S Segment" (Larg e /Small segment) specifies the syst e m where to focus when searching 
images. Please refer to th e similarity to understand the sot of contributing connections, i.e. not 
e v e ry conn e ction is a contributing connection. Small and Large s e gm e nts deploy diff e r e nt scales 
in th e d e t e rmining th e set of conn e ctions. 

Th e "Short/Long" s e arch specifies an imag e sourc e such as whether to s e arch on e directory or 
many dir e ctories. 

Th e "Short Cut" is a Scrollbar to s e l e ct an integ e r b e twe e n 0 and 99; e ach int e g e r is mapp e d to a 
s e t of pred e fin e d settings for th e param e t e rs. 

Th e "Bord e r Cut" is to e liminate the bord e r s e ctions of imag e s. This parameter controls th e 
p e rcentag e of imag e s to b e e liminated befor e entering consideration. 

The "Segm e nt Cut" is b e st illustrat e d by exampl e s. Assume a 1 100x100 imago is reduced to 
100x100 internal repres e ntation, as set by th e param e t e r "Int e rnal R e pr e s e ntation"; th e n 16 
original pix e ls will bo reduced into 1 pixel. Th e n e w valu e of the singl e pixel is determined by 
the parameter "Reduction Type". Th e "Segment Cut' sets a threshold: if the numb e r of non zero 
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pix e ls is gr e at e r than th e thr e shold, th e pix e l will have a non z e ro valu e ; oth e rwis e , th e pix e l will 
have a z e ro valu e . 

Presentation Layer of software for implementation of ABM and APN 
Algorithms 

Figure 2 shows the position of the presentation layer . The presentation layer transforms the 
image data to neural data. The procedure includes: 

1 . Open files from the image source; 

2. Decode the image into pixels arrays; 

3. Process images with a filter; 

4. Reduce the size of images to an internal representation. The users can arbitrarily choose 
the internal representation of the images. Such reduction can be based on individual 
images on a case-by-case reduction, or deploy the same reduction factor across to all 
images. 

5. In the case where many pixels in an image have to be combined into a new pixel before 
leaving this layer, the user can choose a reduction type such as taking average, maximum, 
minimum, or deploy a threshold. The result is an input-vector . 

6. Pass the input-vector imag e array to the next laye r, the ABM or APN layer (See Figure 2 
for the three layers) . 
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ABM Layer of software for implementation of ABM and APN Algorithms 

This Upp e r upper level of this layer has two branches: 

• Training Objects 

• High level training class 

• Low level training class A and 

• Symmetry class 

• Recognition Objects 

• High level recognition class 

• Low level recognition class 

This lower level of this layer has only one class, the memory management class. 

The purpose of the memory management class is to claim memory space from RAM, 64K at a 
time. This memory space will be used for storing the connections. It also returns the unnecessary 
space back to the operating system of the computer. 

The low level training object is to 8 provide all necessary functions used by the high level training 
class. 
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The symmetry object is to implement the symmetry defined earlier. . 

The high level training class incorporates symmetry and implements the ABM or APN 
algorithm. The "image Type" parameter in the user interface will determine which algorithm will 
be used use. 

The basic idea for the training algorithms for both the ABM and the APN is as follows: ( \) Break 
a training image into segments: (2) compute the first connection matrix element from a segment: 
(3) Compute the rest of the matrix elements from the first element: (4) repeat for all segments. 
Breaking an image into segments can be as simple as dividing an image into 10x10 segments. 
For example, a binary image 11111111 can be dividied into 4 segments: 1100000, 00110000. 
00001100. and 00000011. 

ABM Training Algorithm is: 

1 . Delete the existing ABM connections; 

2. Combine an image and its classification into an input vector. 

3. The ABM neural connections are calculated based on the input vector. Let N be the N is the 
number of neurons, these connections can be up to the order of N. The image is randomly 
br e aking broken down into a predefined number of pieces , for example. 10x10 pieces or 8x8 
pieces . 

4. Let an image piece, pi, have K = (kl + k2) pixels, where K is an integer. After imposing the 
pixel vector to the ABM net, kl is the number of neurons excited and k2 is the neurons of 
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neurons grounded. A neural state vector can be constructed to represent such a configuration, 
which kl components being 1 and k2 components being 0. 

5. All neuron configurations together form a space, the configuration space. All neuron 
connections together form a space, the connection space. Now make the configuration vector 
z connection vector = pi, which results in the first synaptic connection matrix element. 

6. Calculations of the rest of the matrix elements is based on a distance in the configuration 
space, either the Hausdorff distance or LI distance or L2 distance, and the first matrix 
element calculated in the last step. The definition of the connection space in the 
TECHNICAL BACKGROUND section yields the definition of a distance between 
connections. Many connection vectors will be in a group with a distance of 1 from pi. Many 
vectors will be in a group with a distance of 2 from pi. A connection represented by pi is 
assigned the largest value. Those connections in the group of distance 1 will have smaller 
values than the first synaptic connection matrix element; those connections in the group of 

distance 2 will have even smaller values After a certain distance, the connection matrix 

elements will be 0. There are many ways to generate these matrix elements from the first 
matrix element. The present invention covers all possible combinations of such a generating 
method. 

5. All such v e ctors togeth e r form a space, th e conn e ction spac e . A distanc e , e ith e r the 
Hausdorff distanc e or LI distance or L2 distanc e can be d e fin e d in this spac e . Such a 
d e finition of a distanc e allows all possibl e connection vectors to b e classifi e d via a distance 
from pi. Many vectors will b e in a group with distance 1 from pi. Many v e ctors will bo in a 
group with distanc e 2 from pi , . . . 
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6. Th e conn e ction r e pr e sent e d by pi is assign e d th e larg e st synaptic connection weight. Those 
conn e ctions in th e distance 1 group will have smaller weights, .... Aft e r a c e rtain distanc e , 
the conn e ction w e ights will b e 0, or th e re will be no conn e ctions. Th e pr e s e nt invention 
cov e rs all possibl e combinations of such a g e n e rating m e thod. 

7. The Markov chain is formed after the connections are established. 

Explanations: 

Stepl. A new neural net is initiated. 

Step 2. The explanation on how to combine an image and its classification into a vector is 
given in the TECHNICAL BACKGROUND section. 

Step 3. The training image is broken into several pieces. For example, an image (10, 20, 
0, 0, 30, 40, 50, 0, 0, 0) can be broken into two pieces, (10, 20, 0, 0, 301 and 140, 50, 0, 
0.01. 

Step 4. For a given image piece, kl are the excited neurons and k2 are the grounded 
neurons. Assume an image is (10, 20, 0, 0, 301, and it is sent to a neural net (0, 1, 2, 3, 
41, then kl = 3 and k2 = 2. The excited neurons are 0, 1, and 4. The grounded neurons 
are 2 and 3. The neuron configuration vector is 1 1001. The neural net configuration space 
is explained in the TECHNICAL BACKGROUND section. 11001 is a point in this 
configuration space. 

Step 5. The neural net connection space is explained in the TECHNICAL 
BACKGROUND section. This step makes the configuration vector and the connection 
vector the same, resulting in calculating the first synaptic connection matrix element. In 
the above example, the configuration vector is 1 1001. Assign it to the connection vector, 
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i.e. the first non-zero connection is between neurons (0, L 41. an u = 10 + 20 + 30. Again, 
both the configuration space and the connection space are explained in the TECHNICAL 
BACKGROUND section. 

Step 6. After the first connection matrix element is calculated, others can be calculated. 
In the above example, the following connection vectors have a distance of 1 from 1 1001 : 
01001. 10001. UIOL 1 1011. and 11000. Thev are obtained bv changing one of the five 
bits. The following connection vectors have a distance of 2 from 11001: 00001. 01101. 

01011.01000. 10101. 10011. 10000. 11111. 11100. 11010 The computation of new 

matrix elements will be determined bv a function f (a. d). where a is the first matrix 
element in Step 5 and d is the distance. For example, f (a. d) = a/(l+d). In the above 
example, apn = 60 and 01001 has the distance of 1 from anu therefore, au = a ou /(l+d)= 
60/(1 +H= 30. Similarly. an dj= aid = 30. 

Step 7. Once the connection matrix is calculated, the ABM neural net is defined. It 
behaves like the Boltzmann Machine, i.e. each neuron has a certain probability to be 
excited, based on the connection matrix. This probabilistic neural net defines a Markov 
chain. 

The following example shows a comparison among an ABM net, an image, the input-vector, the 
first connection vector, the first connection-matrix element value, the rest of the connection 
vectors, and the rest of the connection matrix element values: 

Let a 5-neuron ABM net be (0. 1. 2. 3. 41: 

Let a pixel array of an image segment be: (5. 15. 10. 30. 0. 1. 0. 1. 0 60k 
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Then an input vector is { 10. 20. 0, 0, 30}; 
The first connection vector is: 1 1001 : 

The first connection-matrix element value is: aou= 10 + 20 + 30: 

The rest of the connection vectors are: 01001. 1000L 1 1 101. 1 101 1. and 1 1000: 

The rest of the connection values are: a i 4 «a oi4/n+d)= 60/(1 + 1) = 30. euu= a u = 30. ... 

APN Training Algorithm is: 

1 . Delete the existing ABM connections; 

2. Combine an image and its classification into an input vector. 

3. The ABM neural connections are calculated based on the input vector. Let N be the N4s 
the number of neurons, these connections can be up to the order of N. The image is 
randomly br e aking broken down into a predefined number of pieces. 

4. Let an image piece, pi, have K = (kl + k2) pixels, where K is an integer. After imposing 
the pixel vector to the ABM net, kl is the number of neurons excited and k2 is the 
number n e urons of neurons grounded. A neural state vector can be constructed to 
represent such a configuration, which kl components being 1 and k2 components being 
0. 

5. All neuron configurations together form a space, the configuration space. All neuron 
connections together form a space, the connection space. Now make the configuration 
vector = connection vector = pi. which results in the first synaptic connection matrix 
element. 
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6 - Calculations of the re st of the matrix elements are based on a distance in the 
configuration spac e, either the Hausdorff distance or LI distance or L2 distance, and the 
first matrix eleme nt calculated in the last step. The definition of the connection space in 
the TECHNICAL BAC KGROUND section yields the definition of a distance between 
. connections. Man y connection vectors will be in a group with a distance of 1 from pL 
Many vectors will be in a group with a distance of 2 from pi. A connection represented 
bv pi is assigned the largest value. Those connections in the group of distance 1 will 
have small values: those connections in the group of distance 2 will have even smaller 

values After a certain distance, the connection matrix elements will be 0. There are 

many ways to gen erate these matrix elements from the first matrix element. The present 
invention covers all possible combinations of such a generating method. 

5. All ouch vectors together form a space, the connection space. A distanc e , either tho 
Hausdorff distance or LI distance or L2 distance can be defined in this space. Such a 
definition of a distance allows all possible conn e ction vectors to bo classified via - a 
distance from pi. Many vectors will bo in a group with distance 1 from pi. Many vectors 
will bo in a group with distance 2 from pi, . . . 

6. Tho connection represented by pi is assigned tho largest synaptic connection weight. 

Those connections in tho distance 1 group will have smaller weights After a certain 

distance, tho connection weights will bo 0, or there will b e no connections. Tho prosont 
inv e ntion covers all possible combinations of such a g e nerating m e thod. 

7. The Markov chain is formed after the connections are established. 

8. For each connection, in addition to the synaptic connection weight, a mapping over each 
connection is established. Let kl be a number of neurons in the original kl order 
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connection generated by pi, then this mapping maps from the kl neuron to the kl pixel 
value which excited these neurons. This completes the connection for the original 
segment pi. 

9. The segment, pi, also generated many other connections. If a neuron in this connection is 
one of the original kl neurons in pl 5 then this neuron is mapped into the corresponding 
pixel value, which causes this neuron to be excited; otherwise, this neurons is mapped 
into 0. This completes the mappings of all connections generated by this segment pi. 

The APN Training Algorithm is exactly the same as the ABM Training Alg orithm, with the one 
exception of storing a mapping associated with each connection. Figure 16 show s the ABM and 
APN Training Algorithm. 

Example 

Let a 5-neuron ABM net be (0. 1, 2, 3, 4); 

Let a pixel array of an image segment be: 15, 15. 10, 30, 0. 1, 0. 1. 0 601; 
Then an input vector is (10. 20. 0, 0, 301; 
The first ABM connection vector is: 1 1001 ; 

The first ABM connection-matrix element value is: am* = 10 + 20 + 30; 
The rest of the ABM connection vectors are: 01001. 10001, 11101. llOl Land 11000; 
The rest of the ABM connection values are: au-a nu/fl+d^ 60/(1 + 1) = 30, a^ = au z 
30. ...; 

The first APN connection is (an i A = 10 ± 20 + 30 = 60. (0*10. 1*20, 4 *30)1. where 
(0*10. 1 *20. 4*30) is the mapping for aou: 
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The rest of the APN connections are: {a .a = 30. (l-»20. 4-»3 (m. where (l-»20, 4-»30) 
is the ma pping associate to a n- (fru = 30. (0-»10, 4-»30^. where (0-»10, 4-»30) i s the 
mapping associate with dm ; • • • 

The low-level recognition object is to provide all necessary functions used by the high-level 
recognition class. The high-level recognition class implements the ABM or APN algorithm. The 
"imago Typo" param e tor in tho uoor interface will dotormino which algorithm will bo uogd u se. 

The basic idea for the recognition algorithms for both the ABM and the APN is as follows: 

• A trained ABM or APN net forms a Markov chain, which will settle on an invariant 
distribution; 

• The distribution function will classify the images. 
ABM Recognition Algorithm is: 

1 . An image to be classified is imposed on the Markov Chain. 

2. This Markov chain will settle on its invariant distribution. A distribution function is 
deployed to describe such a distribution. 

3. This distribution function, once obtained, can be used to classify images. This will 
produce triplets of image, class, and weight. Image retrieval and classification are two 
different sides of the same token. 

4. These triplets of image, classification, and weight can be viewed as the results of the 
classification process. For the search process, a doublet of image and weight are 
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displayed. The second part of the triple is omitted because the search problem has only 
one class. 
Explanation 

Step 1. An ABM net is already trained with a training image. II. at this po int. The ABM 
net also behaves like a Markov chain. A new image is entered to the ABM net, which has 
an input vector, x. 

Step 2. This Markov chain has an invariant distribution, described bv a function z = f(v\ 
where v looks like this (x. 0) or (x. IV Here x is an input vector. The d istribution function 
will produce triplets (x. 0. zO) and (x. 1. zlV where zO and zl may or may not be 0. 
Step 3. The invariant distribution function actually classifies the image as follows: the 
probability of x in class 1. meaning x matches the training imap e T1. is zl: and the 
probability of x in class 0. meaning x does not match th e training image II. is zO. The 
ABM recognition algorithm only deals with zl. If zl is greater tha n a certain threshold. 
II and x will be a match: otherwise. II and x will not match. 

Step 4. The triplet (x. 1. zjj is converted into a doublet fx. zlV A sample output is in 
Figure 5. 

APN Recognition Algorithm isi 

1 . An image to be classified is imposed on the Markov Chain. 

2. This chain will settle on its on its invariant distribution. A distribution function is 
deployed to describe such a distribution. 
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3. This distribution function, once obtained, can be used to classify used claooify images. 
This will produce triplets of image, class, and weight. 

4. Comparing the input-vector and the APN-connection-vector modifies this weight. All 
connection vectors together forms a vector space. A distance, either LI distance or L2 
distance can be defined in this space. The basic idea is the new weight will be directly 
proportional to the old weight and inversely proportional to this distance. The present 
invention covers all functions of obtaining the new weight: 

New weight = f (old weight, distance). 
This will produce a new set of triplets of image, classification, and weight. 

5. These triplets of image, classification, and weight can be viewed as the results of the 
classification process. For the search process, a doublet of image and weight are 
displayed. The second part of the triple is omitted because the search problem has only 
one class. 

Explanation 

Step 1. See Step 1 of the ABM Recognition algorithm above. 
Step 2. See Step 2 of the ABM Recognition algorithm above. 
Step 3. See Step 3 of the ABM Recognition algorithm above. 
Step 4. The ABM weight is modified bv a formula. 

New weight = f (old weight distance). 
For example. zV = zl / (1 + distance). The old triplet (x. 1, zlj will beco me a new triplet 
(x. l.zD. 
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Step 5. The triplet (x, IzH is converted into a doublet (x, zV\ A sampl e output is in 
Figure 5. 



IVI API (Image Verification and Identification Application Programming 
Interface) 

A typical imago matching application structur e is: 

• — GUI (graphical us e r interface) Lay e r 

• — DBMS (databaso managem e nt system) Lay e r 

♦ — IVI API (imago verification and identification API) Layer 

« — SPI (Service Provider Intorfaoo) Lay e r 

• — OS (Operating System) and Hardware Lay e r 

The IVI API is transparent for SPI (Sorvioo Provider Interface): the SPI functions will pass right through 
the IVI API. The SPI oan bo accessed directly from layers above tho IVI API layer, i.e. tho DBMS layer 
or GUI layer. 

Thoro are two main functions in API layer: v e rify and identify; and thoro is on e main function in tho SPI 
layer: captur e . 

Tho two top lovol jobs for v e rification are Enrollment and Verify. Tho two top level jobs for 
identification aro Enrollment and Identify. Tho enrollment, in either case, is nothing but sotting a few 
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parameters; tho IVI API deals with tho raw imagos directly. In this API, there is only ono top lovol 
fiinotion for verifications, Verify; and thoro is only on e top lovol function for identifications, Identify. 

This IVI API does not have an enrollment proooss. Tho enrollment is replaced by sotting two parameters: 

• — Tho imag e in question; 

• — Tho folder of previously stored imagos. 

This IVI API do e s r e quire an imago storage structure that should bo followed by tho applications, so th e 
fold e r of previously stored imagos can be pass e d to tho verification and identification functions. 
Both th e verification path and identification path are parameters, whioh can b e changed by tho param e t e r 
writer functions. Th e imago in question can b e stored anywhoro in a hard drive. Tho previously stor e d 
imagos must follow tho following structur e : 

Verification 

Tho previously stor e d imagos must bo stored at: 
■ verification pathVIDV 

Example. Assum e : 

1. Tho v e rification path (a parameter) is: 

o:\Attrasoft\vorification\ 

2. A s e t of doublets is: 
Image imag e lD 
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Ginaljpg 

Gino2.jpg Um± 

Tiffany l.jpg 1^002 

Tiffany2.jpg mQ2 

Then tho storage structure is: 

o:\Attrasoft\vorifioation\12001\ginal.jpg 

o:\Attrasoft\vorifioation\l 200 1 \gina2.jpg 

o:\Attrasoft\vorifioation\ 1 2002\tiffany 1 jpg 

o:\Attrasoft\vorifioation\12002\tiffany2.jpg 

Identification 

Tho fold e r of previously stor e d images must bo stor e d at: 
identification path\ 

Example. Assum e : 

1. Tho identification path (a parameter) is: 

o:\Attrasoft\idontifioation\ 

2. A sot of doubl e ts is: 

Imag e imag e lD 

Ginaljpg 

Gina2.jpg «0W- 

Tiffany ljpg 
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Tiffany2.jpg mQ2 

If the number of images is l e ss than 1000, thon the storago Gtruotur e is 



o:\Attrasoft\idontifioation\ginal jpg 
o:\Attrasoft\idontifioation\gina2jpg 
o:\Attrasoft\idontifioation\tiffany 1 jpg 
o:\Attrasoft\idontification\tiffany2.jpg 

If the number of images is mor e than 1000, thon the sub directories should b e us e d: 

e:\Attrasoft\idontifioation\dir0000\ginal.jpg 
o:\Attrasoft\id e ntifioation\dir0000\gina2.jpg 
o:\Attrasoft\idontification\dir0000\tiffanyljpg 
o:\i\ttrasoft\idontifioation\dir0000\tiffany2jpg 



Enro ll mont 



The enrollment proc e ss builds the folder of previously stored images according to the above struotur e . 
Tho folder of previously stored images will bo a param e ter for the AVI lay e r, called v e rification directory, 
or id e ntification directory or search dir e ctory. There will bo a sootion to addr e ss the paramet e rs lat e r. 
Because tho enrollment m e ans passing parameters, tho enrollm e nt is always 100%. Enrollmont m e ans 
placing images in a folder. For e xampl e , if you want to enroll Gina who, has an ID and an imag e : 
Imag e imag e lD 
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Ginaljpg UMi 

Then you must plaoo the imag e , ginal jpg, to th e oorr e ot fold e r: 

o:\Attrasoft\v e rification\12001\ginal.jpg 
Lator during a verification, if somoono olaim sho is Gina with ID 12001, th e n tho nowly oapturod imag e 
will bo oomparod with the images in this folder: o:\Attrasoft\v e rifioation\12001\. The enrollment rat e is 
always 100% because you can always copy an imago to a fold e r. 



1:N Matching 



Tho following methods (one main function and throe r e sult readers) are used to perform th e Verification 
function: 

int vorify(String imag e , long imagoID); 
long gotVorifyID(); 
String gotVorifyNam e (); 
long gotVorifyWoightQ; 



A typical process is: 
# — Initialize System 
• — Captur e imag e 
• — Calculat e tho templat e 
• — V e rify 
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However, because "Caloulato th e templat e " is not requir e d in this IVI API; and tho systom is initialized 
before th e verification proc e ss start e d, the process is: 

• — Captur e 
• — V e rify 

Th e oaptur e Q functions are provided in SPI, which can bo accessed directly by applications. Both th e 
imago in qu e stion and the fold e r of pr e viously stor e d imag e s ar e in tho hard drive. Th e applications th e n 
pass (String imag e , long imagoID) to th e vorify() function. 

N:N Matching 

Th e following m e thods ore us e d to perform tho Identification function: 
int id e ntify(String imag e ); 
Long [] gotIdontifyID(); 
String [] g e tldontifyNam e Q; 
Long gotldontify Woight(). 

Both th e imago in question and tho fold e r of pr e viously stored imag e s are in tho hard drive. Tho 
applications then pass (String imago) to th e idontifyQ function. 
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.1 



Parameters 



Thoro ar e many param e t e rs us e d in th e algorithm (Figur e . 3), such as Blurring, S e nsitivity, Int e rnal Cut 

Threshold (Ext e rnal Cut) This s e t of paramet e rs tog e ther forms an integer array in tho ohos eft 

programming languag e . Th e s e t of parameters oan bo accessed via th e following sot and g e t Amotions: 

Th e sot of param e t e rs forms an array; 

Void s e tParam e t e r( int I, long x); // a[I] = x 

Long g e tParam e t e r(int I); // r e tun e a[I] 
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Sample Implementation 

We will present three sample implementations based on Figure 2 (3-Layer Architecture). The 
first example has all 3 layers; the second example has only 1 layer; and the third example has 2 
layers. 

There are two CD's labeled "Document, Sample Implementation". The disks contain only three 
ASCII files. Each disk in the duplicate set is identical. The contents of the CD are: 



File Name 


Type 


Size 


Date 


Description 


ABM4_9 


TXT 


156,256 


05-16-02 


Detailed description of ImageFinder 4.9 


ABM5_0 


TXT 


96,515 


05-16-02 


Detailed description of Poly Applet 5.0 


ABM5_1 


TXT 


43,019 


05-16-02 


Detailed description of TransApplet 5.1 



These three files will give detailed descriptions of the three sample implementations below. 
Attrasoft ImageFinder 4.9 

A sample Invention Application Software is the Attrasoft ImageFinder 4.9, which has all three 
layers in Figure 2. Figure 3 shows the ImageFinder User Interface using the Present Invention. 
Figure 4 shows a sample Key Input in the ImageFinder software using the Present Invention. 
Figure 5 shows a sample Search Output of the Present Invention. The search output is a list of 
pairs. Figure 6 shows a sample Classification output of the Present Invention. The classification 
output is a list of triplets. 
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The ASCII file, ABM4_9.TXT, in the CD's labeled "Document, Sample Implementation" will 
give a detailed description. 

In addition, two CD's, labeled "Attrasoft ImageFinder 4.9", contain sample implementation 
software. The software can be installed and run to test the proposed algorithm. Note: 

A. The CD's contain non-ASCII files, such as the installation file and execution files. The 
installation files will install the following executable files to a computer with Microsoft 
Windows as the operating system: 

• Attrasoft ImageFinder 4.9 for Windows 95/98/ME, execution files; 

• Attrasoft ImageFinder 4.9 for Windows 2000/XP, execution files; 

• Data File for running the software; 

• User's Guide in Microsoft Word, and 

• User's Guide in html format. 
These five files can also be run from the CD. 

B. The Operating System is Windows 95, 98, ME, 2000, and XP. 

C. Each disk in the duplicate set is identical. 

D. Contents of the CD. 

Root Directory Contents: 

File Name Type Size Date Description 
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DISK1 


ID 


5 


01-05-90 9*3 Id 


Installation File 


DISK 10 


ID 


5 


01-05-90 9-31d 


Installation File 


DISK1 1 


ID 


5 


01-05-90 9*31d 


Installation File 

XX XkJ li*lll*HV/**. X xx^/ 


DISK12 


ID 


5 


01-05-90 931d 


Installation File 


DISK 13 


ID 


5 


01-05-90 9:32d 


Installation File 


DISK14 


ID 


5 


01-05-90 9 32d 


Installation File 

X.XX4J VMlliiVA Vll X 11V 


DISK2 


ID 


5 


01-05-90 9'32d 


Installation File 


DISK3 


ID 


5 


01-05-90 932d 


Installation File 


DISK4 


ID 


5 


01-05-90 9:33p 


Installation File 


DISK5 


ID 


5 


01-05-90 9:33p 


Installation File 


DISK6 


ID 


5 


01-05-90 9:33p 


Installation File 


DISK7 


ID 


5 


01-05-90 9:33p 


Installation File 


DISK8 


ID 


5 


01-05-90 9:34p 


Installation File 


DISK9 


ID 


5 


01-05-90 9:34p 


Installation File 


SETUP 


EXE 


47,616 


01-05-90 9:3 lp 


Installation File 


SETUP 


INI 


32 


01-05-90 9:3 Id 


Installation File 

11AU VV*1H*V1 VIA X iiV 


SETUP 


INS 


147 ? 449 


01-05-90 9:31d 


Installation File 

XAA|JVWAAV*VAVAA X AAV 


SETUP 


ISS 


510 


01-05-90 9:31p 


Installation File 


SETUP 


PKG 


15,061 


01-05-90 9:3 lp 


Installation File 


JNST32I 


EX_ 


306,666 


01-05-90 9:31p 


Installation File 


ISDEL 


EXE 


8,192 


01-05-90 9:3 lp 


Installation File 


_SETUP 


1 


721,623 


01-05-90 9:3 lp 


Installation File 


_SETUP 


10 


1,454,681 


01-05-90 9:3 lp 


Installation File 
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SETUP 

a \y a 


11 


1 455 574 


01-05-90 9 3 Id 

vi \j»y y\j i u 


Installation File 

xiioicxiitxiivsii x iiv 


SFTT IP 

OJ_> 1 vJX 


12 


i 455 468 


01-05-90 9-31o 


Installation File 

lllO IC111C4.11VJ11 X llv 


SFTT IP 


n 


1 454 113 


01-05-90 9 4 32n 


Installation File 

lllOLCXlltXllvsll X XIV 


SFTT IP 


14 


1 074 165 


01-05-90 9-32o 

\J i v/«»/ ^ v/ y • ~j *-* M 


Installation File 

Alio LCIX1CIIXVS1X A llv 


SFTTJP 


? 
a* 


1 454 796 


01-05-90 9'32d 

\J 1 v/»/ v/ S * *J VJ 


Installation File 

lllk3tC4.llC4.UlV/ 11 A llv 


SETUP 


3 


1 456 887 


01-05-90 9*32d 

wi \J *J y\j y • -} vj 


Installation File 

ilUlUllUllVll 1 XIV 


SETUP 


4 


1 455 245 


01-05-90 933d 


Installation File 


SETUP 


5 


1 455 918 


01-05-90 933d 

v/ J. Vfc/ ✓ v/ y * +j ~j vj 


Installation File 


SFTT IP 


6 


1 455 206 


01-05-90 9-33d 


Installation File 

AXXlJ IwllCIU V/ll A llv 


SETT IP 

OXJ 1 vJ A 


7 


1 453 720 


01-05-90 9'33n 

V/ x V/ «y y\j y »~j-jyj 


Installation File 

A110tC4.llC4-l.lv/ll A llv 


SFTT IP 


8 


1 455 603 


01-05-90 9-34d 

\j x \j *j y\j y • ~j i vj 


Installation File 

141u IMI1MV1 vli X XXV 


SETUP 


9 


1 456 571 


01-05-90 9-34d 


Installation File 


SETUP 


DLL 


10,752 


01-05-90 9:31p 


Installation File 


SETUP 


LIB 


196,219 


01-05-90 9:31d 


Installation File 


ABM49 


<DIR> 


06-08-01 l:04p 


Executable File 


USPT072 


<DIR> 


02-28-01 7:15p 


Data File 


USPT074 


<DIR> 


05-21-01 4:33p 


Data File 



E. Interpretation of the files 

Please see Appendix A for the detailed interpretation of the roles of these files. To install the 
software to a Personal Computer using Windows, double click the setup.exe file. 



Attrasoft PolyApplet 5.0 
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A sample Invention Application Software is the Poly Applet 5.0, which only has the Neural Layer 
of this invention. 

The ASCII file, ABM5_0.TXT, in the CD's labeled "Document, Sample Implementation" will 
give a detailed description. 

Attrasoft TransApplet 5.1 

A sample Invention Application Software is the TransApplet 5.1, which has both Neural Layer 
and the Presentation Layer of this invention. 

The ASCII file, ABM5_1.TXT, in the CD's labeled "Document, Sample Implementation" will 
give a detailed description. 

In addition, two CD's labeled "Attrasoft TransApplet 5.1" contain sample implementation of the 
software library. Note: 

A. The disks contain only Non-ASCII files. The CD contains the following files: 

• Attrasoft TransApplet 5.1 software library for Windows 95/98/ME/2000/XP, 
COM/DLL file format; 

• Sample Implementation Code; 

• User's Guide in Microsoft Word, and 

• User's Guide in html format. 

B. The Operating System is Windows 95, 98, ME, 2000, and XP. 

C. Each disk in the duplicate set is identical. 
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D. Contents of the CD; 



Root Directory Contents: 






File Name 


Type Size 


Date 


Description 


ABM5_1 


DOC 616,448 10-21-01 11:28a User's Guide, Word 


CHAP3 


<DIR> 


10-19-01 4:31p 


Examples 


CHAP4 


<DIR> 


10-19-01 4:31p 


Examples 


CHAP5 


<DIR> 


10-19-01 4:31p 


Examples 


CHAP6 


<DIR> 


10-19-01 4:31p 


Examples 


CHAP7 


<DIR> 


10-19-01 4:32p 


Examples 


FBI 


<DIR> 


06-08-01 l:04p 


Examples 


HELP 


<DIR> 


10-19-01 4:40p 


User's Guide, Word 


OBLIQUE 


<DIR> 


06-08-01 l:04p 


Examples 


README 


TXT 567 


10-20-01 10:51a 


readme.txt 


TRANS-26 


DLL 282,112 


10-21-01 11:00a 


COM DLL 



E. Interpretation of the files 

(El) The file labeled "COM DLL" is the COM DLL software library file to be used by users. 
(E2) The directories, labeled "Examples", contain the examples of how to use the COM DLL. 
(E3) The files, labeled "User's Guide, Word" and the directory, "User's Guide, html", contain 
the User's Guide. 
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CLAIMS 

What is claimed is: 

1 Claim 1 is deleted . 

2 Claim 2 is deleted . 

3 Claim 3 is deleted . 

4 Claim 4 is deleted . 

5 Claim 5 is deleted . 

6 Claim 6 is deleted . 

7 Claim 7 is deleted . 

8 Claim 8 is deleted . 

9 Claim 9 is deleted . 

10 Claim 10 is deleted. 



11 Claim 1 1 is deleted. 

12 Claim 12 is deleted , 

13 Claim 13 is deleted. 

14 Claim 14 is deleted. fCurrontlv amondodl A computer impl e m e nted process of claim 1 and 3. 
wh e r e in a st e ps of specifying param e ters wh e r e in the st e ps of s e tting paramet e rs comprises 
th e S e nsitivity, or whatever th e t e rminology us e d, which defines a distanc e b e tw ee n two 
n e ural ABM n e ts g e n e rat e d by two imag e s in a conn e ction spac e such that the distanc e can 
b e us e d to e liminat e unmatch e d imag e s. A comput e r impl e ment e d proc e ss of claim 1 and 3 
(s e arch and classification), wh e r e in th e st e ps of s e tting parameters comprises th e 
"S e nsitivity", which d e als with th e sample segm e nt siz e , high s e nsitivity is for small 
s e gm e nt(s) and low s e nsitivity is for larg e segm e nt(s). 

15 Claim 15 is deleted. 

16 Claim 16 is deleted. 

17 Claim 17 is deleted. 

18 Claim 18 is deleted. 
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19 Claim 19 is deleted. (Currently am e nd e d) A m e thod of l e arning and recognition of imag e s 
using an ABM l e arning algorithm, an ABM recognition algorithm, an APN l e arning 
algorithm, and an APN recognition algorithm comprising: 



a) training a fully connect e d n e ural n e t with th e ABM l e arning algorithm; 

b) classifying imag e s with th e ABM recognition algorithm via a stabl e distribution 
of an Markov chain; 

c) extending training of a fully conn e ct e d n e ural n e t with th e APN learning 
algorithm from a plurality of a binary n e ural n e ts to multi - valu e d n e ural n e ts; and 

d) e xtending th e ABM recognition algorithm with APN recognition algorithm from a 
plurality of a binary n e ural nets to multi valued neural n e ts wher e in e xt e nding 
furth e r compris e s: 

bl) computing a matching scor e based on th e ABM r e cognition algorithm and a 
distanc e b e tween two images. 

A comput e r impl e m e nt e d proc e ss of claim 1 and 3 (s e arch and classification), wherein the 
neural lay e r d e ploys th e ABM or/and APN algorithm. 

20 Claim 20 is deleted 

21 Claim 21 is deleted. 

22 Claim 22 is deleted. 
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23 Claim 23 is deleted . 



24 Claim 24 is deleted . 

25 Claim 25 is deleted. 

26 Claim 26 is deleted. 

27 Claim 27 is deleted. 

28 Claim 28 is deleted. 
29-49 (cancelled) 

50. (Currently amended) A process for recognizing a digital image using a computer and an 
ABM algorithm, where ABM is the "Attrasoft Boltzmann Machine", which further consists of 
the ABM training algorithm and the ABM recognition algorithm, said ABM training algorithm 
comprises: 

Imposing an image to an ABM so the ABM will be trained, whereas the ABM is a specific 
combination of (a) a fully connected neural network and (b) a Markov chain; 
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Classifying at least one target image based on the invariant distribution function of the trained 
ABM wherein the step of Imposing an image to an ABM further comprises: 

a) deleting existing ABM connections; 

b) creating an input vector, p, based on an input image, x, and its classification, y; 

c) breaking the input vector, p, into a number of pieces, pi, p2, p3..., where such breaking 
could either be logically based on objects/segments or geometrically (such as a division 
of 10 e qual parts) ; 

d) constructing a set of neural state vectors, si, s2, s3 ... according to pi, p2, p3..., whereas 
a state vector, si, has a number of 0's (grounded state) and a number of Ts (excited 
state); all such vectors together form a configuration space, H(P); 

e) computing an initial neural connection from each of pi, p2, p3..., said computation 
comprising: 

el) constructing a connection space, H(C), where each neural connection is a point inside this 
space; 

e2) making the connection space, H(C), from a configuration space, H(C) = 
(H(P), R), where R is a space of real numbers; 

e3) making an initial connection cl to be cl = (pi, 1), or f (pi) = 1, where f(pl) is 
a connection matrix element; 

f) computing the rest of the neural connections from each of the initial connections, cl, c2, 
c3..., said computation comprising: 
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fl) constructing a distance or distances, d (pi, pi'), between an initial neural state, pi, and an 
arbitrary state, pF; said distances can be Hausdorff distance, or LI distance, or L2 distance, 
or any other distances; 

f2) constructing a function, g(d), which maps a distance between two neural vectors, d, to a 
number, g(d); said function comprising of any functions as long as it decreases in value when 
the distance increases , for exampl e , g(d) = 1 / ( 1 + d) or g(d) ~l/(l+d + d*d) ; 
f3) constructing an arbitrary connection element ( pl\ g ( d (pi, pF) ) ) from the initial 
connection element; 

f4) applying ( pl \ g ( d (pi, pF) ) ) for all points in the connection space since the ABM is a 
fully connected network with all possible ranks. 

g) constructing an ABM Markov chain after all of the connections are established. 

51. (cancelled) 

52. (Previously presented) The process of Claim 50, wherein the step of Classifying at least one 
target image based on the invariant distribution function of the trained ABM comprises: 

a) imposing an image to be classified on an ABM Markov chain; 

b) allowing the ABM Markov chain to settle on its invariant distribution, described by a 
distribution function; 
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c) classifying the target image based on this invariant distribution function, said distribution 
comprising of information of classes and weight given by the invariant distribution 
function directly; 

d) presenting the results as a triplet (image, classification, weight), which can be used both 
in image search and image classification. 

53. (Currently amended) A process for recognizing a digital image using a computer and an APN 
algorithm, where APN is "Attrasoft PolyNet", an extension of the ABM, which further consists 
of the APN training algorithm and the APN recognition algorithm, said APN training algorithm 
comprises: 

Imposing an image to an APN so the APN will be trained, whereas the APN is a specific 
combination of (a) a fully connected neural network, (b) a Markov chain, and (c) a mapping 
function (called the APN function); 

Classifying at least one target image based on the invariant distribution function of the trained 
APN wherein the step of Imposing an image to an APN further comprises: 

a) deleting existing APN connections; 

b) creating an input vector, p, based on an input image, x, and its classification, y; 

c) breaking the input vector, p, into a number of pieces, pi, p2, p3..., where such breaking 
could either be logically based on objects/segments or geometrically (such as a division 
of 10 e qual parts) ; 
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d) constructing a set of neural state vectors, si, s2, s3 ... according to pi, p2, p3..., whereas 
a state vector, si, has a number of O's (grounded state) and a number of l's (excited 
state); all such vectors together form a configuration space, H(P); 

e) computing an initial neural connection from each of pi, p2, p3..., said computation 
comprising: 

el) constructing a connection space, H( C ), where each neural connection is a point inside 
this space; 

e2) making the connection space, H( C ), from a configuration space, H( C ) = 
(H(P), R), where R is a space of real numbers; 

e3) making an initial connection cl to be cl = (pi, 1), or f (pi) = 1, where f(pl) is 
a connection matrix element; 

f) computing the rest of the neural connections from each of the initial connections, cl, c2, 
c3..., said computation comprising: 

fl) constructing a distance or distances, d (pi, pi'), between an initial neural state, pi, and an 
arbitrary state, pi'; said distances can be Hausdorff distance, or LI distance, or L2 distance, 
or any other distances; 

£2) constructing a function, g(d), which maps a distance between two neural vectors, d, to a 
number, g(d); said function comprising of any functions as long as it decreases in value when 
the distance increases , for e xampl e , g(d) - 1 / ( 1 + d) or g(d) ~l/(l+d + d*d) ; 
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0) constructing an arbitrary connection element ( pl\ g ( d (pi, pi') ) ) from the initial 
connection element; 

f4) applying ( pi', g ( d (pi, pi*) ) ) for all points in the connection space since the APN is a 
fully connected network with all possible ranks; 

g) constructing an APN Markov chain after all of the connections are established; 

h) constructing an initial mapping to reflect the contribution of the multi-valued neurons; 
said mapping comprising of pairs: (neuron position, input vector value); 

1) constructing the rest of the mappings based on the initial mapping; said mapping 
comprising of pairs: (neuron position, input vector value in the initial mapping). 

54. (Cancelled) 

55. (Previously presented) The process of Claim 53, wherein the step of Classifying at least one 
target image based on the invariant distribution function of the trained APN comprises: 

a) imposing an image to be classified on an APN Markov chain; 

b) allowing the APN Markov chain to settle on its invariant distribution, described by a 
distribution function; 

c) classifying the target image based on this invariant distribution function; said distribution 
comprising of information of classes and weight given by the invariant distribution 
function directly; 

d) constructing intermediate results as a triplet (image, classification, weight); 
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e) computing a new weight, said computation comprising: 

el) constructing a distance between two mappings, the mapping associated with the 
image to be classified and the mapping associated with a connection; 
e2) constructing a function (called the APN function): weight' = h (weight, d); said 
function, h, comprising of any functions as long as it decreases in value when the 
distance, d, increases; 

e3) modifying the old weight by this function; 

f) constructing results as a triplet (image, classification, weight'), which can be used both in 
image search and image classification. 
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Appendix A. Accompanying Application Document and CD 

1. Two CD's Labeled "Document, Sample Implementation" 

A. The disks contain only three ASCII files. 

B. Each disk in the duplicate set is identical. 

C. Contents of the CD: 

File Name Type Size Date Description 

ABM4_9 TXT 1 56,256 05-1 6-02 Detailed description of ImageFinder 4.9 

ABM5J) TXT 96,515 05-16-02 Detailed description of PolyApplet 5.0 

ABM5J TXT 43,019 05-16-02 Detailed description of Trans Applet 5.1 

D. These documents are Copyrighted: 

ImageFinder 4.9: TX 5-385-330 
PolyApplet 5.0: Pending 

Trans Applet 5.1: Pending 

E. Interpretation: These three files will give detailed descriptions of the three sample 
implementations: 

Detailed description of ImageFinder 4.9 
Detailed description of PolyApplet 5.0 
Detailed description of Trans Applet 5.1 

2. Two copies of the coil bound manual labeled "User's Guide, Attrasoft ImageFinder 4.9". 
These are two hard copies of the User's Guide for Attrasoft ImageFinder 4.9. 
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3. Two CD's Labeled "Attrasoft ImageFinder 4.9" 

A. The disks contain only non-ASCII files. The CD contains two types of files: 
(Al) Installation file, which will install the following executable files to a computer with 
Microsoft Windows as the operating system: 

• Attrasoft ImageFinder 4.9 for Windows 95/98/ME, execution files; 

• Attrasoft ImageFinder 4.9 for Windows 2000/XP, execution files; 

• Data File for running the software; 

• User's Guide in Microsoft Word, and 

• User's Guide in html format. 

(A2) A set of executable files running from the CD with Microsoft Windows as the operating 
system: 

• Attrasoft ImageFinder 4.9 for Windows 95/98/ME, execution files; 

• Attrasoft ImageFinder 4.9 for Windows 2000/XP, execution files; 

• Data File for running the software; 

• User's Guide in Microsoft Word, and 

• User's Guide in html format. 

B. The Operating System is Windows 95, 98, ME, 2000, and XP. 

C. Each disk in the duplicate set is identical. 

D. Contents of the CD: 
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Not all contents are listed, because there are too many files. The two data directories, for 
example, contain more than 80,000 files. The Date information is not accurate due to the date 
setting of the computer, which creates these files. 

Root Directory Contents: 



File Name 


Tvoe 


Size 


Date 


Description 


DISK1 


ID 


5 


01-05-90 9:3 lp 


Installation File 


DISK10 


ID 


5 


01-05-90 9:31p 


Installation File 


DISK1 1 


ID 


5 


01-05-90 9:3 lp 


Installation File 


DISK 12 


ID 


5 


01-05-90 9:3 lp 


Installation File 


DISK13 


ID 


5 


01-05-90 9:32p 


Installation File 


DISK14 


ID 


5 ' 


01-05-90 9:32p 


Installation File 


DISK2 


ID 


5 


01-05-90 9:32p 


Installation File 


DISK3 


ID 


5 


01-05-90 9:32p 


Installation File 


DISK4 


ID 


5 


01-05-90 9:33p 


Installation File 


DISK5 


ID 


5 


ai (\ C C\f\ c\ 

01-05-90 9:33p 


T a 11 a* T^'l 

Installation File 


DISK6 


ID 


5 


01-05-90 9:33p 


Installation File 


DISK7 


ID 


5 


01-05-90 9:33p 


Installation File 


DISK8 


ID 


5 


01-05-90 9:34p 


Installation File 


DISK9 


ID 


5 


01-05-90 9:34p 


Installation File 


SETUP 


EXE 


47,616 


01-05-90 9:3 lp 


Installation File 


SETUP 


INI 


32 


01-05-90 9:31p 


Installation File 


SETUP 


INS 


147,449 


01-05-90 9:3 lp 


Installation File 
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SETUP 
SETUP 
JNST32I 

ISDEL 
_SETUP 
_SETUP 
_SETUP 
_SETUP 
_SETUP 
_SETUP 
_SETUP 
_SETUP 
_SETUP 
_SETUP 
_SETUP 
_SETUP 
_SETUP 
_SETUP 
_SETUP 
_SETUP 
ABM49 
USPT072 
USPT074 



ISS 


510 


PKG 

X IVVJ 


15 061 

X \J\J X 


EX 


306 666 


EXE 


8,192 


1 


721 623 


10 

1 v/ 


1 454 681 


11 


1 455 574 


12 


1 455 468 


13 


1 454 113 


14 


1 074 165 


2 


1 454 796 


3 


1 456 887 


4 


1,455,245 


5 


1,455,918 


6 


1,455,206 

A ft I ft w 


7 


1 453 720 


8 


1 455 603 


9 


1 456 571 


DLL 


10,752 


LIB 


196,219 


<DIR> 


<DIR> 


<DIR> 



01-05-90 9:3 lp 
01-05-90 9:31p 
01-05-90 9:31p 
01-05-90 9:3 lp 
01-05-90 9:3 lp 
01-05-90 9:3 lp 
01-05-90 9:3 lp 
01-05-90 9:3 lp 
01-05-90 9:32p 
01-05-90 9:32p 
01-05-90 9:32p 
01-05-90 9:32p 
01-05-90 9:33p 
01-05-90 9:33p 
01-05-90 9:33p 
01-05-90 9:33p 
01-05-90 9:34p 
01-05-90 9:34p 
01-05-90 9:3 lp 

01- 05-90 9:31p 
06-08-01 l:04p 

02- 28-01 7:15p 
05-21-01 4:33p 



Installation File 
Installation File 
Installation File 
Installation File 
Installation File 
Installation File 
Installation File 
Installation File 
Installation File 
Installation File 
Installation File 
Installation File 
Installation File 
Installation File 
Installation File 
Installation File 
Installation File 
Installation File 
Installation File 
Installation File 
Executable File 
Data File 
Data File 
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E. Interpretation of the files 

(El) Files labeled "Installation File" are used to install the following executable files to a 
computer with Microsoft Windows as the operating system: 

• Attrasoft ImageFinder 4.9 for Windows 95/98/ME, execution files; 

• Attrasoft ImageFinder 4.9 for Windows 2000/XP, execution files; 

• Data File for running the software; 

• User's Guide in Microsoft Word, and 

• User's Guide in html format. 

To install the software to a Personal Computer using Windows, double click the setup.exe 
file. 

(E2) The directories, labeled "Data", contain the data files to run the software. There are more 
than 80,000 files in the two directories, so they will not be listed. 

(E3) The directories, labeled "Executable", contain the executable file to run from the CD. 
Directory Contents: 



File Name Type 


Size 


Date Description 


50X50 <DIR> 




06-08-01 l:04p 


Data File 


ABM49_2K EXE 


385,536 


06-08-01 11:54a 


Executable 


ABMAl BMP 


25,182 


08-31-99 11 :02p 


Executable 


DEISL1 ISU 


47,058 


06-08-01 l:05p 


Executable 


IMAGEF-7 <DIR> 




06-08-01 l:04p 


Data File 


JSREG32 DLL 


24,576 


02-07-96 8:07a 


Executable 


ABM48 TXT 


79 


06-08-01 12:26p 


Executable 


ABM48 1 TXT 


6 


03-23-01 9:20a 


Executable 
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ARM48 2 


TXT 


6 


03-23-01 9*21 a 


Kx editable 


ARM48 3 


TXT 


6 


03-23-01 9-21 a 


Executable 

JjAvVUIUUIv 


ABM48 4 


TXT 

X /I X 


6 


03-13-01 2'28d 


Executable 


ABM48 5 


TXT 

X -/ v x 


6 


03-13-01 2-28d 


Executable 


ARM48 6 


TXT 

X V X 


6 


03-13-01 2-28d 


Executable 


ABM48 7 


TXT 

X v X 


6 


03-13-01 2'28d 


Executable 


ABM48 8 


TXT 

X X 


6 


03-13-01 2-29d 


Executable 


ABM48 9 


TXT 

X J. 


6 


03-13-01 2'29d 


Executable 


ABM49 


EXE 


386 048 


06-08-01 10 53a 


Executable 


ABM4_9 


DOC 


3 598 336 


06-07-01 10*52a 


User's Guide, Word 


FBI <DIR> 




06-08-01 1 04o 


Data File 


FINAL 


<DIR> 




06-08-01 1*04d 


Data File 


HELP 


<DIR> 




06-08-01 l'04t> 


User's Guide, html 


L0CK_~15 


<DIR> 




6-08-01 1'04d 


Data File 


L0CK_~17 <DIR> 




06-08-01 l:04p 


Data File 


NOTES 


TXT 


140 


03-23-01 10:33a 


Executable 


OBLIQUE 


<DIR> 




06-08-01 l:04p 


Data File 


STAMP 


<DIR> 




06-08-01 l:04p 


Data File 


USPTO <DIR> 


06-08-01 l:04p Data File 





The interpretation of this subdirectory: 

• The directories, labeled "Data", contain the data files to run the software. 
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• The directory, labeled "User's Guide, html", contains the user's guide files in html 
format. 

• The file, labeled "User's Guide, Word", contains the user's guide files in html format. 

• The files, labeled "Executable", contain the executable files for Windows 
95/98/ME/200/XP. 

4. Two copies of the coil bound manual labeled "User's Guide, Attrasoft TransApplet 5.1" 
These are two hard copies of the User's Guide for Attrasoft TransApplet 5.1. 

5. Two CD's Labeled "Attrasoft TransApplet 5.1" 

A. The disks contain only non-ASCII files. The CD contains the following files: 

• Attrasoft TransApplet 5.1 software library for Windows 95/98/ME/2000/XP, 
COM/DLL file format; 

• Sample Implementation Codes; 

• User's Guide in Microsoft Word, and 

• User's Guide in html format. 

B. The Operating System is Windows 95, 98, ME, 2000, and XP. 

C. Each disk in the duplicate set is identical. 

D. Contents of the CD: 

Root Directory Contents: 

File Name Type Size Date Description 
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arm 1 ; 1 nnr 


616,448 10-21-01 11:28a User's Guide, Word 




CHAP3 


<DIR> 


10-19-01 4:31p 


Exam r>les 


CHAP4 


<DIR> 


10-19-01 4:31p 


Examnles 


CHAP5 


<DIR> 


10-19-01 4:31p 


Examr>les 


CHAP6 


<DIR> 


10-19-01 4:31p 


Examoles 


CHAP7 


<DIR> 


10-19-01 4:32p 


Examples 


FBI 


<DIR> 


06-08-01 l:04p 


Examples 


HELP 


<DIR> 


10-19-01 4:40p 


User's Guide Word 


OBLIQUE 


<DIR> 


06-08-01 l:04p 


Examples 


README 


TXT 567 


10-20-01 10:51a 


readme.txt 


TRANS-26 


DLL 282,112 


0-21-01 11:00a 


COM DLL 



E. Interpretation of the files 

(El) The file labeled "COM DLL" is the COM DLL software library file to be used by users. 
(E2) The directories, labeled "Examples", contain the examples of how to use the COM DLL. 
(E3) The files, labeled "User's Guide, Word" and the directory, "User's Guide, html", contain 
the User's Guide. 

6. Inventor's Resume'. 
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ANNOTATED SHEET 



Enter key image into the system; 




r 


Set training parameters and click the 
training button to teach the system 
what to look for 




f 


Enter search-directory(s) 




f 


Set search parameters), and click the 
search button 




f 



The output is a web page, which is a list 
of names and weights: 

• The weight of an image is 
related to the characteristics you 
are looking for (the weight is 
similar to an Internet search 
engine weight); 

• Click the name of each image 
and an image will pop up on the 
screen. 



Figure 1. 
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Application Layer i 






i 
^ 


i 




Presentation Layer 


i 
1 


i 

f 






\ 


i 
r 


APN 






ABM 



Figure 2. 
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ANNOTATED SHEET 



'\ Attrasoft imageFinder 4.9 



Train Search 



Batch Example Help 



hi 



sent 



Enter a key image 



Search D\t 1 1 Enter a search directory 



mousebpwn at (2891 229 j; 1st point 



{Light Background 128 



Reduction j 


100x100 | 


1 ^B0 


i Symmetry 


Train* 


I Retrain 


Sensitivity:50 


BIurring:5% 


Short 


Shape Cut 


Internal Cut j 


| External Cut 


f^lwiveit^ 


L Segment j 


|[ Search 


f Record - 


Undo | 


Classify _ 


Sort, | 


; Save | 




' Batch (N) 


Batch(t) 


[ FBatch(1) 





[Viewlngl |" File : ^ pll?| 



About 



J Notes M; Hin^Tll tHelp j 



attrasoft 



Figure 3. 
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! Attrasoft ImageFinder 4.9 



Traifi ;Search Classify '.Batch ^xampi^ Help; 



mm 



m 



Key Segment |n Files\Attr^sof^mageFindeAfbiSGodwinaipg \ | Light Background 1 28 



• 



SearcHDir \ |C:\Program Files\AttrasQft\lmageFindef\fbi\. 




mouseD own at (231 . 231 j; 1 st point 

EEC 



lift;: 



Reduction j 


. 100x100 


"360 | 


|[ Symmetfy J 


f train 


;Retrain j 




Blurring: 5% 


iShotfl | 


]j phap^Out j Internal Gut j^xtemal Cut, 


Bi-ievelf | 


t L Segment 


^earch 


Record: 


; Undo ' 


Xlassify 


If Sort j 


Save 


New J 


I BatbH(lsl) 


I Batch(1) 


FBatcH(T) 





Display image! : 



[View Img .} 


File 


j Clear' | -About 


Notes j 


Mints; II Help 


[ attrasoft 



Figure 4. 
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•'Hj C:\aout.htm - Microsoft Internet Explorer 


ismm 






#* * ^> . ® B # 




I llllISI^ C:\aouthtm lS]BM3 




D:\imaselib\fbi\GODWINlA.JPG 24192 




D:\imaeelib\fbi\GODWIN2A.JPG 30400 




D:WeHb\MGOFWW2C.JPG 28928 




D:WeHb\MGODWINA.JPG 33152 


r> ■ 


D:\imafieHb\fbi\GODWINC.JPG 91328 


i 

i 


D:\imaseHb\fbi\GODWINI).JPG 47104 


1 






\mm> t \_wmmm -'4 



Images 
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Figure 5. 
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C:\bout.htm - Microsoft Internet Explorer Q@ 



i$9 ^ d> &r 0} jj| • 'A 
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Figure 6. 
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ANNOTATED SHEET 



Enter key image into the system. 



1 


f 


Set training parameters and click the 
training button to teach the system 
what to look for. 




f 


Enter search-directory(s). 




f 


Set search paramefc 
search button. 


sr(s), and click the 



No 

► 



Start classification and present results: 

• An image link for each image in 
the search directory; 

• The classification weights of 
this image in each search; and 

• The classification of this image 
as a link. 



Figure 7. 

100 




ANNOTATED SHEET 



Provide the batch code to the system. 




r 


Click the batch button to execute the 


code. 





Figure 8. 
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ANNOTATED SHEET 



Combine an image and its classification into a vector. 




r 


All such together form a mathematical configuration space. Each 
point in such a space is called a state. 




r 


A Markov chain exists in such a space where the state of the 
configuration space is a state of the Markov chain. 






The construction of such a Markov chain is by a particular type 
of neural network, called ABM network or APN network. 







The Markov chain will settle on its invariant distribution. A 
distribution function is deployed to describe such a 
distribution. In particular, such a distribution function 
classifies the images. 



Figure 9. 
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Figure 10. 
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Figure 11. 
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ANNOTATED SHEET 



Create an ABM net with no connections 



V 




Combine an image and its classification into an input vector. 






Impose the input vector to the learning module. 






The ABM neural connections are calculated based on the input 
vector. Let N be the number of neurons; the order of 
connections can be up to N and the number of connections can 
be 2**N, where ** represents the exponential function. 




f 


The Markov chain is formed after the connections are 
established. This Markov chain will settle on its invariant 
distribution. A distribution function is deployed to describe 
such a distribution. 




yr 


This distribution function, once obtained, can be used to 
classify images. This will produce triplets of image, class, and 
weight. Image retrieval and classification are two different 
sides of the same token. 




f 


Output results: triplets of image, class, and weight. 



Figure 12. 



f, 

i 



Annotated Sheet 

Create an ABM net with no connections. 

£ 

Combine an image and its classification into an input vector. 



Impose the input vector to the learning module. 



The APN neural connections are calculated based on the input vector. 
Let N be the number of neurons; the order of connections can be up to 
N and the number of connections can be 2**N, where ** represents the 
exponential function. 



A mapping over each connection is established. 



The K-element mapping is changed to N-element mapping. By taking 
the domain of this mapping away, the range of this mapping forms a 
vector, APN connection vector. 



The Markov chain is formed after the connections are established. This 
chain will settle on its on its invariant distribution. 



This distribution function, once obtained, can be used classify images. 



Comparing the input-vector and the APN-connection- vector modifies 
this weight. This will produce a new set of triplets of image, 
classification, and weight. 



Output results: triplets of image, class, and weight. 



Figure 13. 
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Figure 14. 
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(a) 




Figure 15. 
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Create an ABM net with no connections. j 




f 


Combine an image and its classification into an input vector. 







The ABM neural connection matrix is initiated based on the 
input vector. The image is randomly broken down into a 
predefined number of pieces. 



1 


r 


Get the configuration vector for each image piece. 




r 


Make configuration-vector = connection-vector, which results 
in the first synaptic connection matrix element. 




r 


Calculate the rest of the matrix elements via the first matrix 
element and the distances in the connection space. 




f 


The Markov chain is formed after the connections are 
established. 



Figure 16. (a) 
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ABM Learning Algorithm in Figure 16 (a). 




1 


For each connection, add an additional mapping. 



Figure 16. (b) 
Figure 17 . 
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An image to be classified is imposed on the Markov Chain. 




f 



This Markov chain will settle on its invariant distribution. A 
distribution function is deployed to describe such a distribution. 



This distribution function, once obtained, can be used to classify 
images. This will produce triplets of image, class, and weight. 



These triplets of image, classification, and weight can be viewed 
as the result of the classification process. 



Figure 17S. 
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An image to be classified is imposed on the Markov Chain. 




f 


This Markov chain will settle on its invariant distribution. A 
distribution function is deployed to describe such a distribution. 




r 


This distribution function, once obtained, can be used to classify 
images. This will produce triplets of image, class, and weight. 




r 



Let s be the new APN connection vector generated by an input 
search image. Let t be the APN connection vector generated by 
the training vector. A distance between s and t, either LI 
distance or L2 distance, can be calculated. The new weight will 
be directly proportional to the old weight and inversely 
proportional to this distance. The present invention covers all 
functions of obtaining the new weight: 
New weight = f (old weight, distance). 
This will produce a new set of triplets of image, classification, 
and weight. 



These triplets of image, classification, and weight can be viewed 
as the result of the classification process. 



Figure 189. 
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